Enable sharing of schema extensions

[hassec]

Looking at the extra_structures example:

https://github.com/gafusion/omas/blob/14f4a5f57bb04b09919a1bbd14b0c640f9e6092b/omas/examples/extra_structures.py#L14-L37

If one was to save this ODS as json and share it with another person one wouldn't be able to load it with ODS().load() as the extra structures would be unknown.

I couldn't find a solution, is there a recommend way of how such a schema extensions should be shared?

[orso82]

you can use ODS(consistency_check=...).load()

where consistency check can be True False 'warn' 'drop' 'strict' or a combination of those strings (like 'strict_warn_drop')

True: enforce IMAS + extra_structures

False: no enforcement

warn: turn errors into warnings

drop: used with warn to drop keys that would raise a warning

strict: enforce pure IMAS

[hassec]

Thanks for the quick reply @orso82 😊

I would like to use consistency_check=True, but that won't work until the extra_structures are added. So the person I would share a ODS json file with would have to add these extra structures before the load, and I'm not sure what the best way would be to do that.

Should I just save the extra structures as and additional json file, and than have users load that json and call add_extra_structures ?

( I was hoping for something that involved fewer manual steps for a user, e.g. embedding extra_structures somehow in the ODS json)

[orso82]

I see a few options here:

1. Go through the process of asking the extra structures to be added to IMAS
2. have a Python package that sits on top of OMAS, and define the extra structures there
3. manual, as you outlined
4. disable consistency check

for option (2), this is an example from OMFIT:

_extra_structures = {
    'core_profiles': {
        "core_profiles.profiles_1d[:].electrons.rotation.perpendicular": {
            "coordinates": ["core_profiles.profiles_1d[:].grid.rho_tor_norm"],
            "data_type": "FLT_1D",
            "documentation": "electron perpendicular VxB rotation",
            "full_path": "core_profiles/profiles_1d(itime)/electrons/rotation/perpendicular(:)",
            "data_type": "FLT_1D",
            "units": "rad/s",
            "cocos_signal": "TOR",
        },
        "core_profiles.profiles_1d[:].electrons.rotation.diamagnetic": {
            "coordinates": ["core_profiles.profiles_1d[:].grid.rho_tor_norm"],
            "data_type": "FLT_1D",
            "documentation": "electron diamagnetic rotation",
            "full_path": "core_profiles/profiles_1d(itime)/electrons/rotation/diamagnetic(:)",
            "data_type": "FLT_1D",
            "units": "rad/s",
            "cocos_signal": "TOR",
        },
        "core_profiles.profiles_1d[:].ion[:].rotation.perpendicular": {
            "coordinates": ["core_profiles.profiles_1d[:].grid.rho_tor_norm"],
            "data_type": "FLT_1D",
            "documentation": "ion perpendicular VxB rotation",
            "full_path": "core_profiles/profiles_1d(itime)/ion(i1)/rotation/perpendicular(:)",
            "data_type": "FLT_1D",
            "units": "rad/s",
            "cocos_signal": "TOR",
        },
        "core_profiles.profiles_1d[:].ion[:].rotation.diamagnetic": {
            "coordinates": ["core_profiles.profiles_1d[:].grid.rho_tor_norm"],
            "data_type": "FLT_1D",
            "documentation": "ion diamagnetic rotation",
            "full_path": "core_profiles/profiles_1d(itime)/ion(i1)/rotation/diamagnetic(:)",
            "data_type": "FLT_1D",
            "units": "rad/s",
            "cocos_signal": "TOR",
        },
        "core_profiles.profiles_1d[:].ion[:].rotation.parallel_stream_function": {
            "coordinates": ["core_profiles.profiles_1d[:].grid.rho_tor_norm"],
            "data_type": "FLT_1D",
            "documentation": "ion parallel stream function",
            "full_path": "core_profiles/profiles_1d(itime)/ion(i1)/rotation/parallel_stream_function(:)",
            "data_type": "FLT_1D",
            "units": "rad/s"
            # no COCOS transformation
        },
    }
}

# we empty these dictionaries to force a refresh of the structures cache
omas.omas_utils._structures = {}
omas.omas_utils._structures_dict = {}

for _ids in _extra_structures:
    for _item in _extra_structures[_ids]:
        _extra_structures[_ids][_item]['lifecycle_status'] = 'omfit'
    omas.omas_utils._extra_structures.setdefault(_ids, {}).update(_extra_structures[_ids])

[hassec]

Thanks @orso82, will likely stick with 2. for our purposes at least until IMAS is finally open source and there is an easy way to provide such feedback/requests to them.

Thanks again for taking the time to explain & share that example! 🙇‍♂️ Feel free to close this 👍

[orso82]

I agree that providing feedback/requests upstream can be a time consuming process, but at least with OMAS you don't need to wait for IMAS to be open sourced to see the changes in the data dictionary :)

[hassec]

... but at least with OMAS you don't need to wait for IMAS to be open sourced to see the changes in the data dictionary :)

Yep this is great, and is one of the key reasons why we like using OMAS 😉