Snow yaml creation and test for BUFR snocvr file

JCSDA-internal / ioda-converters

Various converters for getting obs data in and out of IODA

9 stars 4 forks source link

Snow yaml creation and test for BUFR snocvr file #1273

Closed YoulongXia-NOAA closed 9 months ago

YoulongXia-NOAA commented 1 year ago

Description

NCEP/EMC obsproc team produced operational snow product named as gdas.tXXz.snocvr.tmXX.bufr_d (XX is 00, 06, 12, 18 Z cycle). The station-based includes some European countries and US and Canada mainly from MADIS. These bufr snow data are supporting the community land snow DA work. A yaml file needs to be created so that bufr2ioda.x can be used to create ioda v3 NetCDF4 file. The created data needs to be checked against original bufr data to ensure that the data conversion is correct. The ctest needs to be passed and yaml file need to be merged into ioda-converter develop branch.

Requirements

The yaml file needs to be created and tested.

Acceptance Criteria (Definition of Done)

Ctest will pass within ioda-bundle and yaml file will be merged into ioda-converter develop branch

Dependencies

YoulongXia-NOAA commented 10 months ago

Obsproc team is figuring out WGOSLID missing issue, if they solve that issue, yaml file can directly handle snocvr snow depth bufr data. If you still feeling that this data converter should be included within ioda-converter not rather than only GDASApp, I can work it later.

ilianagenkova commented 10 months ago

Sorry for the late chime in. There are no plans to merge the adpsfc and snocvr bufr dump files, for Snow DA purposes.

binv /lfs/h1/ops/prod/com/obsproc/v1.1/gdas.20240110/00/atmos/gdas.t00z.adpsfc.tm00.bufr_d type messages subsets bytes NC000000 5 12 2134 2.40 NC000001 314 7896 1288492 25.15 NC000002 133 5455 1244666 41.02 NC000007 27716 50178 10378430 1.81 NC000100 55 2275 442792 41.36 NC000101 1006 31717 5825104 31.53 NC000102 13 98 15794 7.54

binv /lfs/h1/ops/prod/com/obsproc/v1.1/gdas.20240110/00/atmos/gdas.t00z.snocvr.tm00.bufr_d type messages subsets bytes NC000015 3110 11950 1578040 3.84

WIGOS ID info (i.e. the 4 mnemonics, a.k.a 4 WIGOS ID blocks) would only be available for the tanks (aka Message Type) in bold. While the mnemonics are present in the tanks, they are not always populated - that's the data provider duty.

YoulongXia-NOAA commented 9 months ago

@ilianagenkova, thank you for your clarification. If so, my ioda-converter here is useful for snocvr case if WGOSLID is not populated.

YoulongXia-NOAA commented 9 months ago

Python API code has been merged into GDASApp -https://github.com/NOAA-EMC/GDASApp/pull/848. If this data needs for ioda converter in the future, I will open a new issue. I closed this issue now. Thank you all for your help.

BenjaminRuston commented 9 months ago

thanks @YoulongXia-NOAA am I correct that you're saying some decode files in the GDASApp need to be brought into ioda-converters?

@CoryMartin-NOAA and @emilyhcliu would like to just keep you in the loop too for tracking, but would make sense the snow cover is a new file that started recently was in Nov2022 or something like that but greatly increases the data and uses official NOAA pathways

BenjaminRuston commented 9 months ago

and by decode files I mean a yaml and maybe some python

YoulongXia-NOAA commented 9 months ago

@BenjaminRuston, the yaml file cannot replace "missing values --" directly so that yaml file does not work. I am not sure if python API converters within GDASApp can be easily brought to ioda converter when a bufr data is read in. At least I did not see any examples for such a case. Nevertheless, if needed, I need some guidance/thoughts/ideas so that I can work on this dataset.