search

BONSAMURAIS / mojo

The system model frame work. Turning the raw data into into a computational structure.
BSD 3-Clause "New" or "Revised" License
2 stars 1 forks source link

Need support with aggregation to create square SUTs #10

Open agneta20 opened 4 years ago

agneta20 commented 4 years ago

We have now annotated most of the Exiobase data into RDF, however we face some challenges in annotating all the concepts from the ontology. Essentially we need square Supply and Use tables to be able to pick the determining flows when querying the data. I noticed that the mojo system model does something similar. Hence I would appreciate if the developers can support @IKnowLogic with this task.

jakobsarthur commented 4 years ago

Hi @agneta20 @IKnowLogic,

I'm happy to help where I can, but not sure what it is that you need exactly.

agneta20 commented 4 years ago

We need help with getting squared supply and use tables. Exiobase SUTs are non- square 200 product x 164 activities. I know that to develop the HIOT some of the product categories are aggregated (see attached file) to ensure squared matrices. Having squared supply and use tables helps us annotate the data better based on the BONSAI ontology. expecially to identify the primary product (determining flow) from the data.

I noticed that the mojo system model does something similar in addition to an adjustment for the by-product. perhaps you already have the code to produce squared supply and use tables from Exiobase 3.3.17. If you guide us with this it would be great.

Product correspondence_for determining flow.xlsx

mfastudillo commented 4 years ago

[edited nonsense :D]

Note that the products of the HIOT table have a different code, I don't think they even have a URI or if it makes sense to give them a different one.

This can be done with some data manipulation of the matrix. I think it may be better to store a tidy verison of exiobase, on top excel is an awful format for data manipulation.

agneta20 commented 4 years ago

Thanks for the comment...Initially we tried to pick the determining flow from the diagonal as you correctly suggest. However given the supply table in non- square 200 x 164...the diagonal doesnt always correspond to the correct determining flow.

I also agree with the challenges w.r.t HIOT tables and new URIs to be generated.

For now we have a easier solution wher we provide a separate correspondence vector on the product and the corresponding activity discussed here

jakobsarthur commented 4 years ago

@agneta20 if you want maybe it's easiest ot have a quick call to discuss, because as you point out Mojo does exactly this. (THis is part of the system model for which eventually the idea was that the user could define that her-/himself. But with some default option provided. On the mojo github page, Stefano put a documentation with some nice figures explaining the choices.

If you want to work with a tidied up matrix from the get go as @mfastudillo suggested, then it might we worth letting these data manipulations being done by mojo as they already exist.

But if you want we can have a chat via zoom/skype/...

Stefano-MRC commented 4 years ago

@agneta20 Generally speaking, I don't think that you need to make SUTs square in order to define determining products. A determining product can be also selected in a rectangular format. For example, considering Exiobase, an oil refinery plant may have 'motor gasoline' as determining product. When you make the tables square you aggregate products (so you lose information), and the reason for that is just that you want to invert the matrix. Then, once they are square, it is a common practice that on the diagonal there are determining products. Finally, making SUTs square aggregating products is just one way for solving the system of equations, but there are also other techniques that do not change the format of SUTs. This to say that if you aim to be open to any model in the future, I'd suggest you to avoid making 'strong' assumption since the beginning, such as tables must be square. Instead, if you just want to get a initial beta version and then be open in the future to modify your approach, then it is fine. However, we can plan a on-line meeting to talk about as Arthur proposed.

kuzeko commented 4 years ago

Is this is solved by the following? https://github.com/BONSAMURAIS/EXIOBASE-conversion-software/pull/13 https://github.com/BONSAMURAIS/correspondence_tables/pull/29

IKnowLogic commented 4 years ago

@kuzeko They are partial solutions to the problem. The pull requests enable us to correctly identify determining flows using a correspondence table. However, currently, we are not able to output square SUTs. I was thinking we could extend EXIOBASE-conversion-software with a script outputting square SUTs xlsx files from the initial SUTs.

Nevertheless, if the square tables are not needed, the two above mentioned enhancements solves this issue.