[Bug] mpas-jedi ctest fails with #343

[byoung-joo]

Current behavior (describe the bug)

Several mpas-jedi ctests use airs_aqua_obs_2018041500_m.nc4, cris-fsr_npp_obs_2018041500_m.nc4, and iasi_metop-a_obs_2018041500_m.nc4, but #343 did not consider those files.

https://github.com/JCSDA-internal/mpas-jedi/blob/develop/test/testinput/hofx3d.yaml#L157

https://github.com/JCSDA-internal/mpas-jedi/blob/develop/test/testinput/hofx3d.yaml#L175

https://github.com/JCSDA-internal/mpas-jedi/blob/develop/test/testinput/hofx3d.yaml#L193

To Reproduce

What computer are you running on?

Cheyenne

What compilers/modules are you using?

gnu-openmpi, spack-stack v1.4.0

Steps to reproduce the behavior

check out the mpas-bundle, build, run ctest

Expected behavior

All mpas-jedi ctests pass.

Additional information (optional)

[byoung-joo]

The failed log message is attached below.

19: Observer::finalize QC = 
19: AMSUA-NOAA19--hydro brightnessTemperature_1 nlocs = 100, nobs = 100, min = 0, max = 0, avg = 0
19: AMSUA-NOAA19--hydro brightnessTemperature_2 nlocs = 100, nobs = 100, min = 0, max = 0, avg = 0
19: AMSUA-NOAA19--hydro brightnessTemperature_3 nlocs = 100, nobs = 100, min = 0, max = 0, avg = 0
19: AMSUA-NOAA19--hydro brightnessTemperature_15 nlocs = 100, nobs = 100, min = 0, max = 0, avg = 0
19: QC IASI-METOPA brightnessTemperature_16: 100 passed out of 100 observations.
19: QC IASI-METOPA brightnessTemperature_29: 100 passed out of 100 observations.
19: QC IASI-METOPA brightnessTemperature_32: 100 passed out of 100 observations.
19: QC IASI-METOPA brightnessTemperature_35: 100 passed out of 100 observations.
19: QC IASI-METOPA brightnessTemperature_38: 100 passed out of 100 observations.
19: QC IASI-METOPA brightnessTemperature_41: 100 passed out of 100 observations.
19: QC IASI-METOPA brightnessTemperature_44: 100 passed out of 100 observations.
19: ==> destruct MPAS corelist and domain:  0
19: Exception:  source_column:  0
19:     source_filename:    /glade/scratch/bjung/panda-c/code_develop/mpas-bundle/ioda/src/engines/ioda/include/ioda/Variables/Variable.h
19:     source_function:    Variable_Implementation ioda::detail::Variable_Base<Variable_Implementation>::read(gsl::span<U>, const ioda::Selection&, const ioda::Selection&) const [with DataType = int; Marshaller = ioda::detail::Object_Accessor_Regular<int, int>; TypeWrapper = ioda::Types::GetType_Wrapper<int, 0>; Variable_Implementation = ioda::Variable]
19:     source_line:    539
19: 
19: Exception: oops::HofX3D<MPAS, UFO and IODA observations> terminating...
. . .

[gthompsnJCSDA]

Sorry that these files were overlooked. I'll fix them as soon as I can (hopefully today!)

[byoung-joo]

Thanks, @gthompsnJCSDA.

By the way, is there a script to convert sensorScanPosition type from float to integer for already-generated ioda files ? or should we re-process them from bufr (or similar) format ?

[gthompsnJCSDA]

Are you OK with running a Perl script? I created a decent/simple Perl script that can do the job. Otherwise, you really need to code up only 2 commands to fix the files. One step is ncdump into a CDL temporary file in which the variable is changed with a sed substitution; followed by a ncgen command to re-create the final netCDF. My Perl script can step through a simple directory tree, but a more complex set of subdirectories would possibly require more modifications of the Perl code.

[byoung-joo]

Are you OK with running a Perl script? I created a decent/simple Perl script that can do the job. Otherwise, you really need to code up only 2 commands to fix the files. One step is ncdump into a CDL temporary file in which the variable is changed with a sed substitution; followed by a ncgen command to re-create the final netCDF. My Perl script can step through a simple directory tree, but a more complex set of subdirectories would possibly require more modifications of the Perl code.

Thanks, @gthompsnJCSDA. The simple approach using ncdump/ncgen sounds good to me. Let me discuss this with mpas-jedi group. We uses multiple satellite sensors for a month-long period (https://github.com/JCSDA-internal/mpas-jedi/issues/903).

[gthompsnJCSDA]

This discussion item should be consulted for the two commands: ncdump/ncgen that you need to use. https://github.com/orgs/JCSDA-internal/discussions/57