search

pysat / pysatCDAAC

pysat support for CDAAC instruments
BSD 3-Clause "New" or "Revised" License
2 stars 2 forks source link

BUG: some cosmic_gps variables are set without metadata #36

Closed jklenzing closed 1 year ago

jklenzing commented 2 years ago

Describe the bug A number of metadata for cosmic_gps are not set properly. Some (all?) of these are user defined.

To Reproduce See log from recent run: https://github.com/pysat/pysatCDAAC/runs/7616784797?check_suite_focus=true

Expected behavior Metadata should be set correctly. May need to update to use latest pysat standards

Additional context Saving output here for when the log disappears.

pysatCDAAC/instruments/cosmic_gps.py:636: UserWarning: WARNING: missing_value not used since it
  cannot be safely cast to variable data type
    t_list = data.variables[key][:]
cls_instrument_library.py:329: UserWarning: Metadata for variables [OL_par, OL_ipar, OL_vec1, OL_vec2, OL_vec3, OL_vec4] set to defaults, as they were missing in the Instrument.
    test_inst.load(date=date, use_header=True)
test_instruments.py:70: UserWarning: Metadata for variables [MSL_bin_alt] set to defaults, as they were missing in the Instrument.
    idx, idy, = np.where(rem == 0)
pysatCDAAC/tests/test_instruments.py:70: UserWarning: Metadata for variables [MSL_bin_alt, OL_par, OL_ipar, OL_vec1, OL_vec2, OL_vec3, OL_vec4] set to defaults, as they were missing in the Instrument.
    idx, idy, = np.where(rem == 0)
rstoneback commented 2 years ago

Good news is the updated feedback from pysat is working well.

jklenzing commented 1 year ago

For atmprf data, the OL values only have _FillValue as an ncattr, thus the meta data skips these since it only looks for units. See: https://github.com/pysat/pysatCDAAC/blob/7beae1b062f255decf2678659fb9d6b769f75b0e/pysatCDAAC/instruments/cosmic_gps.py#L553-L558

Probably should update this to search for _FillValue as well

jklenzing commented 1 year ago

For atmprf,

for key in keys:
    print(key, data.variables[key].ncattrs())

yields 

Lat ['units', 'valid_range', '_FillValue', 'long_name']
Lon ['units', 'valid_range', '_FillValue', 'long_name']
MSL_alt ['units', 'valid_range', '_FillValue', 'long_name']
Ref ['units', 'valid_range', '_FillValue', 'long_name']
Azim ['units', 'valid_range', '_FillValue', 'long_name']
Pres ['units', 'valid_range', '_FillValue', 'long_name']
Temp ['units', 'valid_range', '_FillValue', 'long_name']
Bend_ang ['units', 'valid_range', '_FillValue', 'long_name']
Opt_bend_ang ['units', 'valid_range', '_FillValue', 'long_name']
Impact_height ['units', 'valid_range', '_FillValue', 'long_name']
Bend_ang_stdv ['units', 'valid_range', '_FillValue', 'long_name']
Bend_ang_conf ['units', 'valid_range', '_FillValue', 'long_name']
OL_par ['_FillValue']
OL_ipar ['_FillValue']
OL_vec1 ['_FillValue']
OL_vec2 ['_FillValue']
OL_vec3 ['_FillValue']
OL_vec4 ['_FillValue']
jklenzing commented 1 year ago

Notes from testing, not every instrument has _FillValue, need to scan multiple possible metadata vals

jklenzing commented 1 year ago

Added a branch downstream of the RC branch for future discussion. The problem is solved locally, but may want to chat in a broader sense about code flow and pysat standards.