search

JCSDA-internal / ioda-converters

Various converters for getting obs data in and out of IODA
9 stars 2 forks source link

update syntax to ioda v2 #1423

Closed BenjaminRuston closed 8 months ago

BenjaminRuston commented 9 months ago

Description

Provide a detailed description of what this PR does. What problem does it fix? What new capability does it add?

will finish long standing tasks for: https://github.com/JCSDA-internal/ioda/issues/379

Issue(s) addressed

Resolves #1422

Checklist

BenjaminRuston commented 9 months ago

believe I can fix this error by uploading a new testoutput but the error seems strange as would have expected this to work, and suggest the file is writing a field with a name rather than an actual group?

PRE-MAIN-INFO BufrParser: Parsing file ./testinput/bufr_sfcshp.bufr
PRE-MAIN-INFO Executing Queries
PRE-MAIN-INFO Building Bufr Data
PRE-MAIN-INFO Exporting Data
PRE-MAIN-INFO Finished [0.022s]
DIFFER : NAME OF GLOBAL ATTRIBUTE : MetaData/dataType : GLOBAL ATTRIBUTE DOESN'T EXIST IN "testoutput/bufr_sfcshp.nc"
DIFFER : NAME OF GLOBAL ATTRIBUTE : dataType@MetaData : GLOBAL ATTRIBUTE DOESN'T EXIST IN testrun/bufr_sfcshp.nc
PatNichols commented 9 months ago

@BenjaminRuston I did an fgrep on the testoutput files and that particular file is the only one that has that global attribute dataType@MetaData even that name as something else. Note the string doesn't appear in the MetaData group but as a global attribute. Probably something from the bufr file?

BenjaminRuston commented 8 months ago

yes this file in the testoutput is not defined correctly:

python3
>>> import h5py
>>> f = h5py.File('bufr_sfcshp.nc', 'r')
>>> f.keys()
<KeysViewHDF5 ['Location', 'MetaData', 'ObsValue']>
>>> f['MetaData']
<HDF5 group "/MetaData" (5 members)>
>>> f['MetaData'].keys()
<KeysViewHDF5 ['datetime', 'depthBelowWaterSurface', 'latitude', 'longitude', 'stationIdentification']>
>>> f.attrs.keys()
<KeysViewHDF5 ['_ioda_layout', '_ioda_layout_version', 'dataType@MetaData']>
>>> f.attrs['dataType@MetaData']
array(['Moored Buoy'], dtype=object)

yes the dataType@MetaData should not be a global attribute key

BenjaminRuston commented 8 months ago

this is ready for review if anyone would like to look it over