search

JCSDA-internal / ioda-converters

Various converters for getting obs data in and out of IODA
9 stars 2 forks source link

[Bug] crash in TEMPO converter #1415

Closed jeromebarre closed 6 months ago

jeromebarre commented 9 months ago

We now have access to the real TEMPO data. I am encountering some issues with the compression in tempo_nc2ioda.py (https://github.com/JCSDA-internal/ioda-converters/blob/develop/src/compo/tempo_nc2ioda.py#L197), see below.

Sometimes, the shapes of lats/lons are different than the shape of time/flg after compression. At other times, all values are being removed (shape=0). This might very well be an issue with the TEMPO data. I can add some checks to skip files where this happens. 

module purge

module use /discover/swdev/jcsda/spack-stack/modulefiles

module load miniconda/3.9.7

module load ecflow/5.8.4

module load mysql/8.0.31

module use /gpfsm/dswdev/jcsda/spack-stack/spack-stack-1.5.0/envs/unified-env/install/modulefiles/Core

module load stack-intel/2022.0.1

module load stack-intel-oneapi-mpi/2021.5.0

module load stack-python/3.10.8

module load jedi-fv3-env

module load ewok-env

module load soca-env

export PYTHONPATH=/discover/nobackup/projects/gmao/geos_cf_dev/cakelle2/JEDI/jcsda/jedi-bundle/build/lib/python3.10:$PYTHONPATH

/discover/nobackup/projects/gmao/geos_cf_dev/cakelle2/JEDI/jcsda/jedi-bundle/build/bin/tempo_nc2ioda.py -i /discover/nobackup/projects/gmao/geos_cf_dev/obs/TEMPO_L2_V01/TEMPO_NO2_L2_V01_20231018T123335Z_S003G01.nc -o [tempo_total.nc](http://tempo_total.nc/) -v 'no2' -c total

Processing variable: nitrogendioxide_total_column

working on /discover/nobackup/projects/gmao/geos_cf_dev/obs/TEMPO_L2_V01/TEMPO_NO2_L2_V01_20231018T123335Z_S003G01.nc

/discover/nobackup/projects/gmao/geos_cf_dev/cakelle2/JEDI/jcsda/jedi-bundle/build/bin/tempo_nc2ioda.py:113: DeprecationWarning: parsing timezone aware datetimes is deprecated; this will raise an error in the future

  time_ref = np.datetime64(AttrData['date_time_string'])

shapes before compression:

lats: (270336,)

flg: (270336,)

shapes after compression:

lats: (265455,)

flg: (268752,)

Traceback (most recent call last):

  File "/discover/nobackup/projects/gmao/geos_cf_dev/cakelle2/JEDI/jcsda/jedi-bundle/build/bin/tempo_nc2ioda.py", line 363, in <module>

    main()

  File "/discover/nobackup/projects/gmao/geos_cf_dev/cakelle2/JEDI/jcsda/jedi-bundle/build/bin/tempo_nc2ioda.py", line 353, in main

    var = tempo(args.input, args.variable, args.column, args.qa_value, args.thin, obsVar)

  File "/discover/nobackup/projects/gmao/geos_cf_dev/cakelle2/JEDI/jcsda/jedi-bundle/build/bin/tempo_nc2ioda.py", line 54, in __init__

    self._read()

  File "/discover/nobackup/projects/gmao/geos_cf_dev/cakelle2/JEDI/jcsda/jedi-bundle/build/bin/tempo_nc2ioda.py", line 229, in _read

    self.outdata[('latitude', 'MetaData')] = lats[flg]

IndexError: boolean index did not match indexed array along dimension 0; dimension is 265455 but corresponding boolean dimension is 268752
jeromebarre commented 9 months ago

For some reasons the converter passes the wrong obs values but I checked the obs value in the raw files directly and they don't seem crazy. See pic attached for the obs values in skylab i.e. in the ioda file: 57ecaa56-b4bc-4c5d-9e54-cf8c63b19d54 Working on it now. This should be easily fixed today or this weekend will add cloud fraction in the data for filtering in UFO.

jeromebarre commented 7 months ago

Solved before the break, will update the converter soon.