Turn C96C48_ufs_hybatmDA and C48mx500_3DVarAOWCDA into a regression test

[DavidNew-NOAA]

Description

This PR is a companion to GDASApp PR #1365 (merged).

It turns C96C48_ufs_hybatmDA and C48mx500_3DVarAOWCDA into a regression test using the JEDI application testing feature. This feature is turned on using the new DO_TEST_MODE parameter added to config.base in GW PR #3115. This parameter is set to "YES" in the yaml defaults for the JEDI-based CI tests in GW.

The motivation for this PR is a need to catch changes in JEDI which alter the outputs of our applications.

Type of change

• [ ] Bug fix (fixes something broken)
• [X] New feature (adds functionality)
• [ ] Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

• Is this a breaking change (a change in existing functionality)? YES
• Does this change require a documentation update? NO
• Does this change require an update to any of the following submodules? YES
  • [ ] EMC verif-global
  • [X] GDAS PR #1365 (merged)
  • [ ] GFS-utils
  • [ ] GSI
  • [ ] GSI-monitor
  • [ ] GSI-utils
  • [ ] UFS-utils
  • [ ] UFS-weather-model
  • [ ] wxflow

How has this been tested?

C96C48_hybatmDA, C96C48_ufs_hybatmDA, C96C48_hybatmaerosnowDA, C48mx500_3DVarAOWCDA, and C48mx500_hybAOWCDA have been tested successfully on Hera

Checklist

• [ ] Any dependent changes have been merged and published
• [X] My code follows the style guidelines of this project
• [X] I have performed a self-review of my own code
• [X] I have commented my code, particularly in hard-to-understand areas
• [ ] I have documented my code, including function, input, and output descriptions
• [X] My changes generate no new warnings
• [X] New and existing tests pass with my changes
• [ ] This change is covered by an existing CI test or a new one has been added
• [ ] Any new scripts have been added to the .github/CODEOWNERS file with owners
• [ ] I have made corresponding changes to the system documentation if necessary

[DavidNew-NOAA]

@CoryMartin-NOAA Done!

[DavidNew-NOAA]

I misunderstood how default yamls work. I'm merging Cory's branch into mine and redoing the test references

[RussTreadon-NOAA]

g-w CI tests on Hera

The C96C48_ufs_hybatmDA 20240224 00Z gfs_atmanlvar job failed due to the reference check

 0: OOPS_STATS Run end                                  - Runtime:    751.37 sec,  Memory: total:   632.81 Gb, per task: min =     6.27 Gb, max =     7.60 Gb
 0: Run: Finishing oops::Variational<FV3JEDI, UFO and IODA observations> with status = 0
 0: terminate called after throwing an instance of 'oops::TestReferenceFloatMismatchError'
 0:   what():  Test reference Float mismatch @ Line:2
 0: Test Val : 5.6385719460046617e+05
 0: Ref  Val : 6.0590797038184456e+05
 0: Delta    : 4.2050775781378383e+04
 0: Relative tolerance: 5.8488258249115540e+02
 0: Absolute tolerance: 1.0000000000000001e-05
 0: Test Line: 'CostJo   : Nonlinear Jo(Aircraft) = 5.6385719460046617e+05, nobs = 476911, Jo/n = 1.1823111536543844e+00, err = 2.2308811279218053e+00'
 0: Ref Line : 'CostJo   : Nonlinear Jo(Aircraft) = 6.0590797038184456e+05, nobs = 504555, Jo/n = 1.2008759607611550e+00, err = 2.2437239435917040e+00'
srun: error: h35m09: task 0: Aborted (core dumped)

The gfs and gdas use different data dumps. The gdas analysis assimilates more data than the gfs.

We need two reference check files - one for gdas_atmanlvar and another for gfs_atmanlvar. If we don't want to test gfs_atmanlvar, we need to change the yaml used by the gfs_atmanlvar job.

[DavidNew-NOAA]

Thanks, @RussTreadon-NOAA. I'll look into how to fix this