add a generic gnss ro bufr decoder to ioda

[BenjaminRuston]

Description

create a generic GNSS RO BUFR decoder to IODA this depends on the eccodes library and uses a simple python interface

Requirements

new feature: this uses the eccodes python interface along with the ioda python engine to generate a IODA file

Acceptance Criteria (Definition of Done)

ability to generate a IODA file

[fcvdb]

I conducted the following test:

1) Download COSMIC WMO BUFR file: bfrPrf_C2E4.2020.349.23.56.R17_0001.0001_bufr (2020/12/14 23:56) from CDAAC

2) Convert the WMO BUFR file from BUFR to IODA2 using the command gnssro_bufr2ioda.py utility:

ioda-bundle/iodaconv/src/gnssro/gnssro_bufr2ioda.py -d 2020121500 -i bfrPrf_C2E4.2020.349.23.56.R17_0001.0001_bufr -o bfrPrf_C2E4.2020.349.23.56.R17_0001.0001_bufr.ioda2.nc4

3) Convert the WMO BUFR file into NOAA BUFR using cdaac2ncep.py utility (using CDAAC package: ).

./cdaac2ncep.py 2020.350 6 cosmicrt

4) Convert the NOAA BUFR into IODA1 using the Fortran converter with the command:

ioda-converters/bin/gnssro_bufr2ioda 2020121500 cosmicrt.gpsro.t00Z.20201215.bufr cosmicrt.gpsro.t00Z.20201215.ioda1.nc4

5) IODA1 to IODA2 convertion using the command:

ioda-bundle/bin/ioda-upgrade.x cosmicrt.gpsro.t00Z.20201215.ioda1.nc4 cosmicrt.gpsro.t00Z.20201215.ioda2.nc4

6) Dump the two IODA2 obs files into text with ncdump. The listings are attached.

It appears that the gnssro_bufr2ioda.py is missing the following:

1. the ndatetime dimension, but this is now obsolete and not needed anymore.
2. the metadata section (required).
3. the preQc section (required).

bfrPrf_C2E4.2020.349.23.56.R17_0001.0001_bufr_iod [cosmicrt.gpsro.t00Z.20201215.bufr_ioda2.txt](https://github.com/JCSDA-internal/ioda-converters/files/7705274/cosmicrt.gpsro.t00Z.20201215.bufr_ioda2.txt) a2.txt

[fcvdb]

To use the converter gnssro_bufr2ioda.py with the eccodes:

On Orion, load Python module:

module load python/3.9.2

Create a virtual environment:

python3 -m venv eccodes ~/eccodes_env

Active the virtual environment:

source ~/eccodes_env/bin/activate

Install the necessary modules:

python3 -m pip install --upgrade pip setuptools sdist wheel

Install ecmwflibs

python3 -m pip install ecmwflibs Install eccodes module:

python3 -m pip install eccodes

Check encodes install:

python3 -m eccodes selfcheck

Run a test:

cp /home/bruston/work/eccodes/examples/decode_gnss_ro.py . cp /home/bruston/work/eccodes/examples/data/bfrPrf_C2E4.2020.349.23.56.R17_0001.0001_bufr .

Edit decode_gnss_ro.py and set up the INPUT variable, then run:

python3 decode_gnss_ro.py

Deactivate the virtual environment:

deactivate

You may have to set-up the environment variable:

export ECCODES_DIR=/home/bruston/work/eccodes/eccodes-2.23.0-Build

[srherbener]

I've got eccodes working on my MacBook (BigSur) using jedi-stack and pip install. Turns out that Mark Olah had started a build script for eccodes in jedi-stack.

[climbfuji]

I've got eccodes working on my MacBook (BigSur) using jedi-stack and pip install. Turns out that Mark Olah had started a build script for eccodes in jedi-stack.

@srherbener Great news! Do you think you can create a branch for my PR that updates the Python builds, or do you want to create a follow-up PR?

[srherbener]

There are two pieces to eccodes: A compiled library from ECMWF and a pip install for the python API.

I'll start a new PR to build the compiled library (which will be a pre-requisite for the pip install). The we can add the 'pip install eccodes' to your PR (or a subsequent PR).

[srherbener]

I've got the gnssro_bufr2ioda.py script successfully writing a file that contains the MetaData group with its associated variables. I pushed a commit to the feature/generic-gnss-ro-decoder-to-ioda branch with this fix.

[fcvdb]

I updated the script with Stephen's latest commit and converted again file: bfrPrf_C2E4.2020.349.23.56.R17_0001.0001_bufr

I have uploaded the dump from both the Fortran converter (unchanged) and the Python converter (after Stephen's changes). The listing are attached.

In the Global attribute section, gnssro_bufr2ioda.py converter is missing:

     :date_time = 2020121500 ;

In the MetaDatasection, gnssro_bufr2ioda.py is missing the following variables:

   float altitude(nlocs) ; 
   int ascending_flag(nlocs) ;
   float impact_height(nlocs) ;
   float sensor_azimuth_angle(nlocs) ;  
   float time(nlocs) ;                  

The variable impactParameter should be impact_parameter

bfrPrf_C2E4.2020.349.23.56.R17_00 [cosmicrt.gpsro.t00Z.20201215.bufr_ioda22.txt](https://github.com/JCSDA-internal/ioda-converters/files/7707982/cosmicrt.gpsro.t00Z.20201215.bufr_ioda22.txt) 01.0001_bufr.ioda22.txt

[srherbener]

Thanks @fcvdb for testing the new converter output.

We have deprecated the global attribute date_time which is why I removed it. It is part of a deprecated datetime representation. I can certainly put it back in if you are using it for some other purpose.

For the actual netcdf file that you created to form the listing of the expected variables, does this file also contain the correct values of the variables. I'm thinking that I could take that netcdf file and install it as the test reference for this feature branch and then the idea will be to get the script output to match that file. If you agree that this makes sense, please send me the netcdf file.

Also, I will need help in determining how to generate the missing variables from the input bufr file. Ie, which mnemonics produce the missing variables, any formulas, etc.

Thanks!

[srherbener]

I've renamed impactParameter to impact_parameter in the ioda output file.

For the missing metadata variables (eg, MetaData/altitude) it looks like there is a dictionary defined in the eccodes API that gives you access through a codes_get_array function. I have no idea how this works, so I will need instructions on how to get the missing variables out of eccodes.

[huishao-r]

@BenjaminRuston Seems the merged PR didn't close this issue. Would you like to close it?