gnssro_bufr2ioda.py crashes on GRAS, GRACE and Eumetsat Commercial corrupted WMO BUFR files

[fcvdb]

The ioda-converters/src/gnssro/gnssro_bufr2ioda.pycrashes in some instances of corrupted input WMO BUFR files from GRAS, GRACE and Eumetsat Commercial data. This happens when a variable is missing in the file. For example:

GRACE file: iodaconv/src/gnssro/gnssro_bufr2ioda.py -> missing airTemperature
GRAS file: gras_mta_d20210806_s010638_r14183_ekmi.bfr -> Missing second in the datetime string
Eumetsat Commercial file: spi_d20220901_s000300_e001300_iuti.bfr -> Missing height

The occurences are rare. However, because the gnssro_bufr2ioda.pyconverts by batch of input files (~1000), all the other non-corrupted files are not converted when one of those corrupted files is present in the batch.

The ideal solution would be that the converter have the capability to skip the corrupted data while processing the batch. Alternatively, a (fast) function to screen the files integrity prior to run the converter could be applied.

[fcvdb]

I copied an example of the corrupted GAS, GRACE and Eumetsat Commercial WMO BUFR files on Orion in:

/work2/noaa/jcsda/fvandenb/data/data_from_nrl_bufr

The commands to test each file are (Assuming ioda-bundle/iodaconv has been built beforehand in directory <build> and the PYTHONPATH set up:

export PYTHONPATH="<build>/lib/python3.10:${PYTHONPATH}"

cd <build>/iodaconv/test
../../bin/gnssro_bufr2ioda.py -q -r 1 -d 2021080606 \
-i  /work2/noaa/jcsda/fvandenb/data/data_from_nrl_bufr/gras_mta_d20210806_s010638_r14183_ekmi.bfr \
-o gras_mta_d20210806_s010638_r14183_ekmi.nc4 \
&> gras_mta_d20210806_s010638_r14183_ekmi.out

../../bin/gnssro_bufr2ioda.py -q -r 1 -d 2021080606 \
-i  /work2/noaa/jcsda/fvandenb/data/data_from_nrl_bufr/grace_d20210806_s012751_edzw.bfr \
-o grace_d20210806_s012751_edzw.nc4 \
&> grace_d20210806_s012751_edzw.out

../../bin/gnssro_bufr2ioda.py -q -r 1 -d 2021080606 \
-i  /work2/noaa/jcsda/fvandenb/data/data_from_nrl_bufr/spi_d20220901_s000300_e001300_iuti.bfr \
-o spi_d20220901_s000300_e001300_iuti.nc4 \
&> spi_d20220901_s000300_e001300_iuti.out

[BenjaminRuston]

@fcvdb is this getting close, did you want me to look at anything

[BenjaminRuston]

@fcvdb is there an update on this,, is this completed

[fcvdb]

This issue was resolved through PR#1430 option to retrieve only bending angle also add sub sat id . Closing for now.