Metadata to indicate data is infilled

[Rlamboll]

I'm suggesting to put a metadata label on every row returned by the silicone function so that we can easily add them to another database without confusing what is original data and what is constructed. What are people's thoughts on this?

[gidden]

@znicholls can you advise on how this kind of op is handled in openscm?

[byersiiasa]

I think this makes sense - because I dont think including it in the variable name is an option

[gidden]

@byersiiasa agreed. @Rlamboll is the idea here to use pyam? if so, metadata is keyed on model/scenario combinations, rather than variables. I can think of a few options of how to handle it, but thought @znicholls might want to chime in first.

[Rlamboll]

Yes, we want to return a pyam database. So I imagine that mostly people will want to infill all of the data for a given gas, hence we would want model/scenario/gas detail, although potentially people would only infill for some time-period and then add to a pre-existing data.

[znicholls]

@znicholls can you advise on how this kind of op is handled in openscm?

we just add an extra column to our metadata table so we just add an extra column as needed. https://github.com/openclimatedata/openscm/blob/077f9b5db182fcfe791e46f55acd38761e4865eb/openscm/scmdataframe/base.py#L871

If I were doing it, I'd just add a column infilled to data which would indicate if/how the data was infilled. It could be dropped before being used/published but would be helpful for keeping track of things as the data flowed through the steps.

I think OpenSCM's metadata handling is quite different to pyam (we just have one big meta dataframe where we hold everything that isn't values rather than relying on model-scenario keys) so I'm not sure if this solution would actually work or not in pyam?

[Rlamboll]

This will now be handled by iiasa-climate-assessment to streamline the process