test_gdasapp_util_viirsaod2ioda fails on Hera Rocky 8

[RussTreadon-NOAA]

When GDASApp develop at 2198b41 is built and run on Hera Rocky 8 nodes, test_gdasapp_util_viirsaod2ioda fails with

1395: terminate called after throwing an instance of 'oops::TestReferenceFloatMismatchError'
1395:   what():  Test reference Float mismatch @ Line:4
1395: Test Val : 2.2290000000000001e-03
1395: Ref  Val : 2.1952299999999998e-03
1395: Delta    : 3.3770000000000241e-05
1395: Relative tolerance: 2.2121149999999997e-09
1395: Absolute tolerance: 0.0000000000000000e+00
1395: Test Line: '    Min: 0.002229'
1395: Ref Line : '    Min: 0.00219523'
1/1 Test #1395: test_gdasapp_util_viirsaod2ioda ...Subprocess aborted***Exception:   2.29 sec

The Hera Rocky 8 execution of this test does not generate a result consistent with the test reference.

This issue is opened to document this failure and its resolution.

[CoryMartin-NOAA]

Looking more closely at this, it's not just the minimum that varies. There are significant (non-trivial) differences. I will try to investigate and figure out what is going on, but it's more than just relaxing tolerances.

For example: Orion:

1396: time_coverage_end type 2018-04-15T04:19:58Z
1396:  thinthreshold 0
1396:  channels [4]
1396:  number of channels 1
1396:  eigen... row and column:10 10
1396:  total location 40
1396: Processing files provided by VIIRSAOD
1396: Reading... viirs_aod_2.nc4
1396: row,col 1010
1396: time_coverage_end type 2018-04-15T04:19:58Z
1396:  thinthreshold 0
1396:  channels [4]
1396:  number of channels 1
1396:  eigen... row and column:10 10
1396:  total location 100
1396:  appending: viirs_aod_2.nc4
1396:  obs count: 140
1396: Test     : Reading: [viirs_aod_1.nc4,viirs_aod_2.nc4]
1396: Test     : seconds since 1970-01-01T00:00:00Z
1396: Test     : obsVal:
1396: Test     :     Min: 0.00219523
1396: Test     :     Max: 0.163349
1396: Test     :     Sum: 11.8086
1396: Test     : obsError:
1396: Test     :     Min: 0.00885619
1396: Test     :     Max: 0.130309
1396: Test     :     Sum: 8.75665
1396: Test     : preQc:
1396: Test     :     Min: 0
1396: Test     :     Max: 2
1396: Test     :     Sum: 105
1396: Test     : longitude:
1396: Test     :     Min: 141.747
1396: Test     :     Max: 142.872
1396: Test     :     Sum: 19953.1
1396: Test     : latitude:
1396: Test     :     Min: 50.2839
1396: Test     :     Max: 51.3225
1396: Test     :     Sum: 7140.63
1396: Test     : datetime:
1396: Test     :     Min: 1523765998
1396: Test     :     Max: 1523765998
1396: Test     :     Sum: 213327239720

and Hera

1395: time_coverage_end type 2018-04-15T04:19:58Z
1395:  thinthreshold 0
1395:  channels [4]
1395:  number of channels 1
1395:  eigen... row and column:10 10
1395:  total location 40
1395: Processing files provided by VIIRSAOD
1395: Reading... viirs_aod_2.nc4
1395: row,col 1010
1395: time_coverage_end type 2018-04-15T04:19:58Z
1395:  thinthreshold 0
1395:  channels [4]
1395:  number of channels 1
1395:  eigen... row and column:10 10
1395:  total location 66
1395:  appending: viirs_aod_2.nc4
1395:  obs count: 106
1395: Test     : Reading: [viirs_aod_1.nc4,viirs_aod_2.nc4]
1395: Test     : seconds since 1970-01-01T00:00:00Z
1395: Test     : obsVal:
1395: Test     :     Min: 0.002229
1395: Test     :     Max: 0.163349
1395: Test     :     Sum: 9.52747
1395: Test     : obsError:
1395: Test     :     Min: 0.00885619
1395: Test     :     Max: 0.129295
1395: Test     :     Sum: 6.13876
1395: Test     : preQc:
1395: Test     :     Min: 0
1395: Test     :     Max: 1
1395: Test     :     Sum: 37
1395: Test     : longitude:
1395: Test     :     Min: 141.747
1395: Test     :     Max: 142.872
1395: Test     :     Sum: 15097.7
1395: Test     : latitude:
1395: Test     :     Min: 50.2839
1395: Test     :     Max: 51.3225
1395: Test     :     Sum: 5396.7
1395: Test     : datetime:
1395: Test     :     Min: 1523765998
1395: Test     :     Max: 1523765998
1395: Test     :     Sum: 161519195788
1395: Writing ioda file

the obs counts differ...

[CoryMartin-NOAA]

I think I know what the problem is, preqc is missing from the YAML but does not warn/fail. For whatever reason, it used to assume a value of 2 but now assumes a value of 1. Probably somewhere in eckit. I will issue a PR being explicit ASAP.