search

JCSDA-internal / ioda-converters

Various converters for getting obs data in and out of IODA
9 stars 4 forks source link

Ioda-converters YAML for [3D]-RTMA ADPUPA BUFR #1454

Closed PraveenKumar-NOAA closed 8 months ago

PraveenKumar-NOAA commented 8 months ago

Description

This PR adds an ioda-converters YAML for the [3D]-RTMA ADPUPA BUFR data.

The following files are new/ modified:

testinput/bufr_ncep_rtma_adpupa.yaml testinput/rtma.t00z.adpupa.tm00.bufr_d testoutput/rtma.t00z.adpupa.tm00.nc CMakeLists.txt

The ioda-validation test is performed and following results were obtained:

Final results: errors: 0 warnings: 0

The validation of the output ioda obs is also performed in: https://github.com/JCSDA-internal/ioda-converters/issues/1451

Issue(s) addressed

Resolves #1450

Dependencies

None

Impact

None

Checklist

nicholasesposito commented 8 months ago

Looks great

fcvdb commented 8 months ago

@PraveenKumar-NOAA The Netcdf output file size is a little bit large (2.48Mb), is there a way you could subsample the input BUFR file to reduce the size of the output? Thanks.

PraveenKumar-NOAA commented 8 months ago

@PraveenKumar-NOAA The Netcdf output file size is a little bit large (2.48Mb), is there a way you could subsample the input BUFR file to reduce the size of the output? Thanks.

@fcvdb I used only one subset for the BUFR file and which is less than 1 Mb. I am not sure if it can be reduced further. Similar is true for for prepBUFR PR.

PraveenKumar-NOAA commented 8 months ago

@fcvdb @PatNichols I fixed the file sizes, thanks to @ADCollard.

BenjaminRuston commented 8 months ago

thank you @PraveenKumar-NOAA I've run this on a file like this which would be obtained from either the NCAR RDA archive or the NOMADS server and it completed fine and produced a reasonable looking file:

gdas.adpupa.t00z.20220216.bufr

file: 20220215T21_PT6H_adpupa_nesdis_bufr.nc4
range(/MetaData/dataReceiptTime): [--, --]
range(/MetaData/dateTime): [2022-02-15 21:00:00, 2022-02-16 02:00:00]
range(/MetaData/latitude): [-90.0, 82.5]
range(/MetaData/longitude): [-170.7100067138672, 179.22000122070312]
range(/MetaData/stationElevation): [-22, 4508]
range(/MetaData/stationIdentification): [02527, ZVQEQCM]
range(/MetaData/stationWIGOSId): [, ]
range(/ObsValue/airTemperature): [185.85000610351562, 307.1499938964844]
range(/ObsValue/dewPointTemperature): [163.25, 300.1499938964844]
range(/ObsValue/pressure): [300, 109900]
range(/ObsValue/windDirection): [0, 360]
range(/ObsValue/windSpeed): [0.0, 156.0]
range(/QualityMarker/airTemperature): [14, 14]
range(/QualityMarker/dewPointTemperature): [14, 14]
range(/QualityMarker/pressure): [14, 14]
range(/QualityMarker/stationElevation): [1, 1]
range(/QualityMarker/windSpeed): [14, 14]

one thing I notice is the pressure variable is not in MetaData which has been standard practice with radiosonde . It is instead under the ObsValue

@gthompsnJCSDA and @CoryMartin-NOAA and @emilyhcliu we should discuss think we can move forward for now, but believe our current practice is to have pressure under MetaData as it is essentially the location information

BenjaminRuston commented 8 months ago

getting this plot for the above file for observation only: image

gthompsnJCSDA commented 8 months ago

FYI. This is already merged but the ObsTeam needs to do a better job with validating the incoming data against the IODA-validate program. The dimension name RadiosondeReportLevelData is ad hoc and never before seen. I realize we have been lax with ioda-validate set to ignore warnings and errors, but it's up to our own team and reviewers to know that we cannot allow new entires to go against the conventions or else we have no point in the conventions.

Let's be sure to correct the dimension name and MetaData/pressure in place of ObsValue/pressure in a subsequent PR please.