search

JCSDA-internal / ioda-converters

Various converters for getting obs data in and out of IODA
9 stars 4 forks source link

create IODA2 converter for concatenated raob BUFR data from LDM server #722

Closed BenjaminRuston closed 2 years ago

BenjaminRuston commented 2 years ago

Description

create an ecCodes based IODA2 converter for raob BUFR files from an LDM server

Requirements

this requires raob BUFR data and the ecCodes library and the python ecCodes interface

Acceptance Criteria (Definition of Done)

successful production of IODA2 input file

Dependencies

ecCodes must be an option as well as the associated python interface

srherbener commented 2 years ago

I've got the concatenation working for multiple messages now. I doubled the ctest data by cat'ing the file with the single bufr message with itself.

The ctest passes, which has a perfect match up between the two messages in the test input file (since they are copies). I coded the concatenator to extend a variable that is missing from the new message with missing values. This function hasn't been tested yet.

I haven't tested with the full file yet. I'll do that next. I just wanted to get the to you so you can try it out.

After the full file, I'll work on reading multiple files.

srherbener commented 2 years ago

The converter that is checked into github works on the 2020 Dec 15 file: 2020121500.sonde.BUFR. This was the file that the eccodes bufr_count program didn't like. This file contains ~1.7M locations.

I'm working on processing a list of files in one call to the python script.

srherbener commented 2 years ago

I've pushed a commit to github that has the multiple input file processing working. I added a ctest for this where the '-i' argument just lists the WMO bufr file twice.

The 'i' option will accept a space separated list of one or more input bufr files. This should allow you to use the shell file globbing mechanism (eg -i *.sonde.BUFR).