cmutel
commented
7 months ago @fjuniorr First of, wow! This is awesome analysis, and it's fantastic to bring in other systems. I feel very lucky that you have agreed to work with us on this.
Using FlowMap might be the right choice. But I would have put some of the examples differently, sorry for not being clearer in my documentation or code.
For me, the beauty of randonneur is that is doesn't impose a schema - rather, it just says that if you start with a set of matching conditions, based on the "natural" serialization of an object, then you should apply some attribute changes to make that object compatible with the second system. So I would have written your example:
transformation_spec = {
"update": [
{
"source": {
"sumi": "80596FA1-6D62-4392-AC71-509E5F73D39E",
},
"target": {
"name": "Zinc, in ground",
"uuid": "be73218b-18af-492e-96e6-addd309d1e32",
"context": ["natural resource", "in ground"],
},
}
]
}I would add the mapper, datetime, and memo to a metadata section applicable to a set of transformations.
The unit conversion is trickier; I need more time to think about that.
We can't rely on SimaPro having unique identifiers, we will need to match to name, context, and possibly unit. Here is a typical line we would need to harmonize from a SimaPro CSV export:
Emissions to air
Ethane, 1,1,1-trifluoro-, HFC-143a;;g;0,00043524;Undefined;0;0;0;Refrigerant emissions (R-143a, 52% of R-404a), based on PEF guidance documentUpon some reflection, I think you might be right that there is an implicit schema, but I don't that is avoidable... Do we have an example of an OpenLCA flow mapping file in the wild?
fjuniorr
tfardet
msrocka
The
randonneurdata migration verbs are focused on specifying the changes that need to be made in order to convert a source object into a target object.In our case the focus is converting one source flow to a target flow that may differ in some characteristics but otherwise represent the same flow[^20231117T094934].
[^20231117T094934]: From (Edelen, et al., 2017)
Take for example this one-to-one mapping[^20231117T090313] (suffix in ground):
[^20231117T090313]: Using this simapro excel sheet represented as a dict and this encoinvent xml represented as a dict using
xmltodict.One way to express the conversion using
randonneurupdateverb is:Which would generate
I think there are two main downsides to this approach.
Firstly, because we are specifying actual conversions rules, we need to impose a schema on the data. In this example I'm using GLAD flow mapping format but the problem persists even if the target flow list is always the same (eg. ecoinvent) because the problem will appear when a client application (ie. LCA software) expects to consume data in a different format.
Secondly, a conversion that keeps all the metadata from the target flow is verbose because all the fields need to be specified.
A more flexible approach so that we don't need to impose a schema on the target flow is to encode matching information on the target node and not transformations. Extra metadata for the transformations needed (such as unit conversions) should also be added but the client application should do the actual conversion. For example[^20231117T093941]:
[^20231117T093941]: It should be noted that probably makes more sense to reuse openLCA schema FlowMap then to create a new format. This means that the input and output flow lists need to be standardized.
In the randonneur examples it's model of making changes to data is not so problematic because both source and target data already share the same schema.
The challenge of how to generate the mapping information in a way that is reproducible and inspectable for individuals that don't code remains.