search

gafusion / omas

Ordered Multidimensional Array Structure
http://gafusion.github.io/omas
MIT License
30 stars 15 forks source link

Problem with loading ods that has been previously saved in hdf5 or ascii formats #154

Closed pankin closed 3 years ago

pankin commented 3 years ago

I have saved an ods in hdf5 and ascii formats from omfit using the following commands:

geq=OMFIT['EFIT']['FILES']['gEQDSK']
peq=OMFIT['EFIT']['FILES']['pFILE']
ods=ODS(imas_version='3.30.0')
ods_peq = OMFITpFile.to_omas(peq, ods=ods, gEQDSK=geq)
save_omas_h5(ods_peq, ods=ods, "/home/pankin/transp/DIIID/174819/3000/efitfiles/174819_p.h5")
save_omas_ascii(ods_peq, "/home/pankin/transp/DIIID/174819/3000/efitfiles/174819_p.dat")

I tried to open these two files from a python session outside OMFIT. However, there was a problem

In [1]: import omas                                                                                                            

In [2]: ods = omas.load_omas_ascii("/home/pankin/transp/DIIID/174819/3000/efitfiles/174819_p.dat", imas_version='3.30.0')      
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
~/.local/lib/python3.8/site-packages/omas/omas_core.py in __setitem__(self, key, value)
    812             try:
--> 813                 structure = imas_structure(self.imas_version, location)
    814                 if isinstance(value, ODS):

~/.local/lib/python3.8/site-packages/omas/omas_utils.py in imas_structure(imas_version, location)
    792             for key in path[1:]:
--> 793                 structure = structure[key]
    794         _ods_structure_cache[imas_version][ulocation] = structure

KeyError: 'centroid'

During handling of the above exception, another exception occurred:

LookupError                               Traceback (most recent call last)
<ipython-input-2-27aaeca0aca1> in <module>
----> 1 ods = omas.load_omas_ascii("/home/pankin/transp/DIIID/174819/3000/efitfiles/174819_p.dat", imas_version='3.30.0')

~/.local/lib/python3.8/site-packages/omas/omas_ascii.py in load_omas_ascii(filename, machine, pulse, run, dir, consistency_check, imas_version)
    256             continue
    257 
--> 258         ods[path] = value
    259 
    260     return ods

~/.local/lib/python3.8/site-packages/omas/omas_core.py in __setitem__(self, key, value)
    992         if len(key) > 1:
    993             try:
--> 994                 self.getraw(key[0])[key[1:]] = pass_on_value
    995             except LookupError:
    996                 if dynamically_created:

~/.local/lib/python3.8/site-packages/omas/omas_core.py in __setitem__(self, key, value)
    992         if len(key) > 1:
    993             try:
--> 994                 self.getraw(key[0])[key[1:]] = pass_on_value
    995             except LookupError:
    996                 if dynamically_created:

~/.local/lib/python3.8/site-packages/omas/omas_core.py in __setitem__(self, key, value)
    992         if len(key) > 1:
    993             try:
--> 994                 self.getraw(key[0])[key[1:]] = pass_on_value
    995             except LookupError:
    996                 if dynamically_created:

~/.local/lib/python3.8/site-packages/omas/omas_core.py in __setitem__(self, key, value)
    992         if len(key) > 1:
    993             try:
--> 994                 self.getraw(key[0])[key[1:]] = pass_on_value
    995             except LookupError:
    996                 if dynamically_created:

~/.local/lib/python3.8/site-packages/omas/omas_core.py in __setitem__(self, key, value)
    841                             options = list(numpy.array(list(options.keys()))[index[-5:]][::-1]) + ['...']
    842                         options = 'Did you mean: ' + ', '.join(options)
--> 843                     raise LookupError(underline_last(txt, len('LookupError: ')) + '\n' + options)
    844 
    845         # check what container type is required and if necessary switch it

LookupError: Not a valid IMAS 3.30.0 location: equilibrium.time_slice.0.profiles_1d.centroid
                                                                                    ^^^^^^^^
Did you mean: f_error_index, q_error_index, elongation_error_index, elongation, surface_error_index, ...
smithsp commented 3 years ago

I looked at this with @pankin and didn't see any obvious mistakes.

orso82 commented 3 years ago

@pankin the issue is that equilibrium.time_slice.0.profiles_1d.centroid is not a proper location in the IMAS data dictionary. At this point IMAS does not have a placeholder for the centroid of the equilibrium flux surfaces. This is something that we have added temporarily to propagate that information through ODSs, while we either (1) open a JIRA issue, and ask for that information to be added to the IMAS data dictionary (2) find another way to propagate that information (for example, store the flux surfaces information, and then re-calculate the centroid where needed). The addition to the IMAS dictionary is done here: https://github.com/gafusion/OMFIT-source/blob/unstable/omfit/omfit_classes/fluxSurface.py#L3133-L3178

Fixing this in a more proper way is something that we have already discussed with @TimSlendebroek

The consistency_check argument/attribute handles how to treat these cases.

to replicate your issue in OMFIT

ods=OMFIT['tmp'].to_omas()
ods.save('tmp.h5')

ods1=ODS(consistency_check='strict') # `strict` says to discard all `extra_structures` additions
ods1.load('tmp.h5')

To avoid adding extra structures to your ODS:

ods=OMFIT['tmp'].to_omas()
ods.consistency_check='strict_drop_warn' # this will drop the `extra_structures` and warn about doing so
ods.save('tmp.h5')

ods1=ODS(consistency_check='strict') # `strict` says to discard all `extra_structures` additions
ods1.load('tmp.h5')

To avoid loading extra structures to your ODS:

ods=OMFIT['tmp'].to_omas()
ods.save('tmp.h5')

ods1=ODS(consistency_check='strict_drop_warn') # this will drop the `extra_structures` and warn about doing so
ods1.load('tmp.h5')