Update global_enkf ctest for tapering of perturbations (#806)

CatherineThomas-NOAA commented 1 week ago

Description

The global_enkf ctest needs to be updated to include a change to the configuration of GFSv17. The upper layer ensemble spread is too large, so we are enabling a new feature in the EnKF to reduce the spread as part of the ensemble analysis. This feature was previously included in the GSI EnKF source code (Issue https://github.com/NOAA-EMC/GSI/issues/728) and being turned on in the global-workflow through https://github.com/NOAA-EMC/global-workflow/pull/3097. The g-w PR is not required to exercise this PR.

Fixes #806

Type of change

[x] New feature (non-breaking change which adds functionality)

How Has This Been Tested?

The taperanalperts feature was tested in an ATM-only C384/C192 cycling parallel on Hera. The GSI regression tests were run on Hera (results in following comment).

Checklist

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] New and existing tests pass with my changes
[x] Any dependent changes have been merged and published

CatherineThomas-NOAA commented 1 week ago

Hera ctest results

Test project /scratch1/NCEPDEV/da/Catherine.Thomas/gsi/feature_taper/build
    Start 1: global_4denvar
    Start 2: rtma
    Start 5: hafs_3denvar_hybens
    Start 4: hafs_4denvar_glbens
    Start 6: global_enkf
    Start 3: rrfs_3denvar_rdasens
1/6 Test #3: rrfs_3denvar_rdasens .............   Passed  1216.42 sec
2/6 Test #4: hafs_4denvar_glbens ..............***Failed  1880.73 sec
3/6 Test #6: global_enkf ......................   Passed  1950.77 sec
4/6 Test #1: global_4denvar ...................   Passed  2568.77 sec
5/6 Test #5: hafs_3denvar_hybens ..............   Passed  6740.22 sec
6/6 Test #2: rtma .............................   Passed  13935.89 sec

83% tests passed, 1 tests failed out of 6

Total Test time (real) = 13935.93 sec

The following tests FAILED:
      4 - hafs_4denvar_glbens (Failed)

The hafs_4denvar_glbens test failed due to runtime:

The runtime for hafs_4denvar_glbens_loproc_updat is 297.647849 seconds.  This has exceeded maximum allowable threshold time of 294.302768 seconds,
resulting in Failure time-thresh of the regression test.

A rerun of the hafs test succeeded:

Test project /scratch1/NCEPDEV/da/Catherine.Thomas/gsi/feature_taper/build
    Start 4: hafs_4denvar_glbens
1/1 Test #4: hafs_4denvar_glbens ..............   Passed  1408.14 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) = 1408.18 sec

I will run the tests on additional machines tomorrow.

RussTreadon-NOAA commented 1 week ago

@CatherineThomas-NOAA , who would you like to request two peer reviews from?

CatherineThomas-NOAA commented 1 week ago

@TravisElless-NOAA and @AndrewEichmann-NOAA: Would you be available to review this PR?

CatherineThomas-NOAA commented 1 week ago

From the Hera ctest global_enkf_hiproc_updat stdout:

 TAPERANALPERTS  = T,
 TAPERANALPERTS_AKBOT    =   500.000000000000     ,
 TAPERANALPERTS_AKTOP    =  -1.00000000000000

For the effect of tapering itself, I usually use the spread netcdf files produced from the epos step. Using the data from the ctest, I tried to look at the differences between the increments of a couple members, but it's hard to tell if there's an effect there. The tapering does not set spread to zero at the model top, just reduces it.

For previous comparisons when taperanalperts is turned on, see these slides

CatherineThomas-NOAA commented 1 week ago

Dogwood ctests

Test project /lfs/h2/emc/da/noscrub/catherine.thomas/gsi/feature_taper/build
    Start 1: global_4denvar
    Start 2: rtma
    Start 3: rrfs_3denvar_rdasens
    Start 4: hafs_4denvar_glbens
    Start 5: hafs_3denvar_hybens
    Start 6: global_enkf
1/6 Test #3: rrfs_3denvar_rdasens .............   Passed  728.50 sec
2/6 Test #6: global_enkf ......................   Passed  852.35 sec
3/6 Test #2: rtma .............................   Passed  969.81 sec
4/6 Test #5: hafs_3denvar_hybens ..............   Passed  1153.78 sec
5/6 Test #4: hafs_4denvar_glbens ..............   Passed  1214.00 sec
6/6 Test #1: global_4denvar ...................   Passed  1683.33 sec

100% tests passed, 0 tests failed out of 6

RussTreadon-NOAA commented 1 week ago

This PR can be scheduled for merger once we get a second approval from a peer reviewer.

@AndrewEichmann-NOAA , can you review this PR?

NOAA-EMC / GSI

Update global_enkf ctest for tapering of perturbations (#806) #807