Transfer selected blocks from global_variables.yml to case metadata

[perolavsvendsen]

As discussed off-line in various contexts. There is a need for consumers to have access to selected configuration elements, alias-lists, etc which is typically stored in various configuration elements within the FMU setup.

In fmu-dataio today, we already fetch some known blocks from the global_variables.yml and inject them into both case and object metadata, i.e. masterdata.

The suggestion here is to inject more such known blocks and put them into the case metadata when this is produced. Consumers can then fetch these elements from the case metadata when using FMU results.

Suggested pattern to use:

• fmu-dataio has list of known and selected blocks from global_variables
• When making case metadata, fmu-dataio will extract the known blocks if they exist and inject them into the case metadata under a specific tag (suggest to call it config). In case metadata, this could look something like this (stub):

masterdata:
  xx
class: case
tracklog:
  xx
config:
  block1:
    <as in global_variables>
  block2:
    <as in global_variables>

• The blocks will not be made searchable in Sumo - the intention is for the consumer to grab the case metadata and do what it needs to from there.
• If a known block does not exist, fmu-dataio shall move on (if it is missing in the outgoing case metadata, this means that it is missing in the global_variables, so there is consistency)

Things to consider

• We need to be very careful wrt producing case metadata which validates against the schema. Currently, if case metadata fails on validation, case will not be registered, but FMU workflow will still continue. No subsequent uploads will work (because case is not registered in Sumo).
• We must include config in the schema (as a type object). Also include the individual blocks?
• White-listing vs open? Due to the current state of the global_variables (and configs in general) we cannot simply include everything from the global_variables.yml. Therefore, some form of white-listing must be done. However, we could consider explicitly including individual blocks into the schema and not allow for other properties. (Currently, whitelisting of e.g. data.content is not done in via the schema, but via code in fmu-dataio, but this is inconsistent and possibly confusing.)

Needed discussions and inputs

• Give explicit use case(s) to solve in first iteration.

[anders-kiaer]

Three important pieces of metadata with example use cases:

Well attributes/metadata

For each well in the model we have metadata like

• alias (what is it called in RMS, what is it called in Eclipse, what is in called in PDM...)
• type of well

The use case of the alias is to link different data sources (e.g. Eclipse with PDM or RMS). The use case for well type is to be used in e.g. well filters.

An example of file existing in the wild today is e.g. well_attributes.json which can be exported from RMS if I recall correctly (@lindjoha). The current format has some limitations I think:

• "name" is AFAIK RMS name (maybe natural in a file exported from RMS, but not so natural when the metadata is stored in SUMO and read without RMS as main context). Should a more official well name be used as "name" perhaps in the SUMO case metadata?
• if multi-segment wells are modelled as separate wells in Eclipse/RMS, we do not currently have a link between these two in the file (they would have separate RMS and Eclipse names I guess). Should have a more official name at top level with the 1+ well segments beneath?
• wells can change type during their lifetime. Currently it is only one type per well.

FIPNUM vs region/zone In order to combine different data sets, we would need a mapping between FIPNUM (in Eclipse) and region/zone. The current format used in Webviz is like:¨

REGION:
    WestLowland:
    - 1
    - 8
    - 15
    CentralSouth:
    - 2
    - 9
    - 16
    CentralNorth:
    - 3
    - 10
    - 17
    NorthHorst:
    - 4
    - 11
    - 18
    CentralRamp:
    - 5
    - 12
    - 19
    CentralHorst:
    - 6
    - 13
    - 20
    EastLowland:
    - 7
    - 14
    - 21
  ZONE:
    Valysar:
    - 1
    - 2
    - 3
    - 4
    - 5
    - 6
    - 7
    Therys:
    - 8
    - 9
    - 10
    - 11
    - 12
    - 13
    - 14
    Volon:
    - 15
    - 16
    - 17
    - 18
    - 19
    - 20
    - 21

I.e. selecting the region WestLowland and all zones would correspond to FIPNUM = 1 + 8 + 15. Selecting region WestLowLand and zone Valysar would correspond to FIPNUM = 1 (the intersection of the two lists).

Stratigraphy model information This metadata is used to answer e.g. what are the subzones of a given zone. A two-level example (could be higher I guess in real life) is shown here (the file is exported from RMS I think @lindjoha?). The color information is not crucial I think, but we have got positive user feedback wrt. them seeing the same colors in Webviz for the zones as they have set it up in RMS when modelling.

---

There is another final piece of metadata wrt. zone mapping and that is zone mapping vs model layer z-indices. E.g. some file encoding the information shown here. This metadata perhaps/probably have to live on realization level though since it can vary?

[perolavsvendsen]

Ok but these are not part of the global_variables.yml? Case metadata is generated at startup of a case, and we do not alter it afterwards. E.g. if some content is made during runtime, we cannot put that into case metadata.

[perolavsvendsen]

Made a first attempt 👆. What immediately strikes me is the apparent overlap/difference between different configs. I don't think any standard exists here, which means that the road might be a little bit longer. But unsure...

It is relatively straight-forward to fetch contents from global_variables.yml since this already is passed to fmu-dataio when creating case metadata. However, passing other "random" config files that exists is more tricky, especially without breaking changes. So I suggest we stick to global_variables.yml for now (without exactly knowing the consequences of that).

[lindjoha]

The format of well_attributes.json was agreed with Emerson and the idea was that they would make a job to export it. But currently it's just a hard-coded script. I agree with your comments about the limitations, but I am not sure if we want time dependent well attributes? Maybe it's better that a well f.ex can be both "oil producer" and "gas injector". MLTs should definitely be exported as one well.

The script that exports stratigraphy.json is also hard coded, but also there the idea was that we could export the zonation used in RMS. Any zone can have subzones so the format is completely flexible that way. The color parameter is optional, but it is very nice to be able to control the colors.

[perolavsvendsen]

Regarding well_attributes.json and stratigraphy.json:

• Can they go into case metadata?
  • Are they static to the case? (do not change, do not differ between realizations, etc)
  • Are they premade (available when we make the case metadata, e.g. stored in the template project, not made during runtime)?
  • Are their locations in the template project standard and predictable?*

*) The hook workflow that makes case metadata runs as a PRE_SIM hook workflow, and is currently a bit limited in terms of how many arguments it should take, and so on. Alternatively, it needs its own configuration with pointers - but then we are quickly exploding the scope, and we need to figure out if it is sustainable and maintainable.

[lindjoha]

1. They should be static yes. Stratigraphy is definitely static. I guess we can imagine cases where well attributes are not, f.ex if a well is turned into an injector based on some criteria. But it was intended to be static when exported from RMS.

2. They can be premade, but the plan behind exporting them from RMS in the FMU workflow is that updates will automatically be exported.

3. They have been put in rms/output/wells and rms/output/zone, but it can be discussed if they should go into the share folder.

[perolavsvendsen]

stratigraphy is not the same STRATIGRAPHY as we already have in global_variabels.yml? Can they be merged, perhaps?

[perolavsvendsen]

2. They can be premade, but the plan behind exporting them from RMS in the FMU workflow is that updates will automatically be exported.

That makes sense. But in that case, it cannot go into the case metadata. So then we might need "something else". Either export these things are regular data objects/files (with metadata), or we might have to look into supporting some kind of "realization metadata". The downside is that it increases the scope quite a lot, and must involve ERT and others.

[lindjoha]

stratigraphy is not the same STRATIGRAPHY as we already have in global_variabels.yml? Can they be merged, perhaps?

They might, but if I remember correctly, the stratigraphy in global_variables is referring to horizons instead of zones? This was sometimes a bit confusing when dealing with zones and subzones.

[perolavsvendsen]

It covers both. It is a list of dictionaries, where the RMS-name of a horizon or zone is the key. Example from Drogon:

The challenge with the current STRATIGRAPHY could be that the key is the RMS-spelling of the horizon/zone. The name attribute denotes the actual spelling in masterdata. It is currently primarily used to translate between the two, but can be expanded to include other things as well.

The STRATIGRAPHY block from global_variables.yml should not be a bit problem to include in the case metadata, I think. So if we can expand on that, rather than having two separate "stratigraphy" keywords, that would be better I think.

[lindjoha]

Makes sense. Can you add subzones in a tree-structure?

[perolavsvendsen]

What are subzones in this context? This dict should include all zones defined in the model, irrespective of them being in the strat column or not (then they will simply have stratigraphic: false) in them.

[lindjoha]

In RMS it looks like this. The names in Column_1 is a higher level, that could be f.ex Jurrassic or whatever. The json-format is completely flexible with respect to this.

[perolavsvendsen]

Ah... yes, OK, then I see what you mean. Currently, it would be Horizons and Column (zones). The only mechanism currently in use that kind of supports this today is the stratigraphic_alias tag. E.g. Top Valysar is the same as "Top TopVolantis_BaseVolantis" (the zone-naming in Drogon is a bit strange).

I also assume that the JSON starts on the right side of the chart, not the left. E.g the root level is "column_1"?

May not be straight-forward to merge them if they are fundamentally different on the root level. Do you have an example of the resulting JSON?

[lindjoha]

Yes, the json starts from the right, and the levels in the tree is flexible. Any zone can have subzones. Here is how the json file looks for the Drogon example:

https://github.com/equinor/webviz-subsurface-testdata/blob/master/01_drogon_ahm/realization-0/iter-0/rms/output/zone/stratigraphy.json

[perolavsvendsen]

OK so in order to represent both into the same structure (case metadata, or case metadata + global_variables.yml), we must either:

• Rename one or both, and use both. Would probably be relatively doable, but creates a mess (perhaps)?
• Normalize one to the other. Would probably require quite fundamental refactoring somewhere, and probably some backwards compatibility issues.

The term stratigraphy is so generic that it is bound to cause conflicts...

I think this would require some sort of offline discussion at some point. It is a relatively big topic. So far I think we can conclude that simply putting (the other) stratigraphy into global_variables.yml is not straight forward.

[perolavsvendsen]

Discussion topics 2023-05-24:

• Cases vs ensembles. Sometimes, pred is part of the same case as iterations while sometimes, pred runs (restarts) are separate cases. pred will typically have different e.g. well attributes than iterations.
• Metadata can vary across realizations in some cases.

Well attributes:

• List of all wells with categorical attributes, comes from RMS. Currently hardcoded as a dictionary within RMS.
• Currently (partly?) stored in RMS. Webviz needs access to these metadata. They may vary across iterations/ensembles. They are technically model metadata (?) - but the model may change across iterations.

Stratigraphy:

• Several "stratigraphy" exists. The one based on RMS stratigraphy is generated in RMS during GUI runs. This can be exported to disk, and picked up by e.g. dataio.
• Can this file be exported during runtime by RMS?
• Do not have access to the multi-level stratigraphy in RMS.
• But why use RMS as a "truth" here, when SDMA should technically be the one source of definitions?
• What about the zonation used in the simulator? This belongs to the (generated) grid and will differ between realizations.

Configs that are static to the case: can be included in case metadata

• Stratigraphy, currently in global_variables
• Stratigraphy, currently used by Webviz, coming from RMS

Configs that are NOT static to the case (either vary across iteration or realization): must be exported per realization

• Well attributes
• Zonation used in the simulator

Can we export everything per realization, then figure out afterwards if they are identical?

To be continued...