Add WE2E verification capability

[chan-hoo]

Description

• Add the WE2E verification option to the main tasks.

Anticipated changes to regression tests:

• [ ] Is baseline change expected ?

Subcomponents involved:

• [ ] apply_incr.fd (NOAA-PSL/land-apply_jedi_incr)
• [ ] ufs_model.fd (ufs-community/ufs-weather-model)
• [ ] none

Linked PR's and Issues:

Resolves Issue #130

Testing (for CM's):

• RDHPCS
  • [x] Hera
  • [x] Orion
  • [ ] Hercules
• CI
  • [x] Completed
• PW-Clouds
  • [ ] AWS
  • [ ] AZURE
  • [ ] GCP

[chan-hoo]

ctest:

Test project /scratch2/NAGAPE/epic/Chan-hoo.Jeon/landda_we2e/land-DA_workflow/sorc/build
    Start 1: test_create_ens
1/4 Test #1: test_create_ens ..................   Passed    3.79 sec
    Start 2: test_letkfoi_snowda
2/4 Test #2: test_letkfoi_snowda ..............   Passed   16.22 sec
    Start 3: test_apply_jediincr
3/4 Test #3: test_apply_jediincr ..............   Passed    1.81 sec
    Start 4: test_ufs_datm_land
4/4 Test #4: test_ufs_datm_land ...............   Passed   44.66 sec

100% tests passed, 0 tests failed out of 4

Total Test time (real) =  66.49 sec

[chan-hoo]

• WE2E verification log file (we2e.log):

  20000104.000000 :: ANALYSIS :: sfc_data.tile1 :: TEST was complete successfully.
  20000104.000000 :: ANALYSIS :: sfc_data.tile2 :: TEST was complete successfully.
  20000104.000000 :: ANALYSIS :: sfc_data.tile3 :: TEST was complete successfully.
  20000104.000000 :: ANALYSIS :: sfc_data.tile4 :: TEST was complete successfully.
  20000104.000000 :: ANALYSIS :: sfc_data.tile5 :: TEST was complete successfully.
  20000104.000000 :: ANALYSIS :: sfc_data.tile6 :: TEST was complete successfully.
  20000104.000000 :: ANALYSIS :: xinc.tile1 :: TEST was complete successfully.
  20000104.000000 :: ANALYSIS :: xinc.tile2 :: TEST was complete successfully.
  20000104.000000 :: ANALYSIS :: xinc.tile3 :: TEST was complete successfully.
  20000104.000000 :: ANALYSIS :: xinc.tile4 :: TEST was complete successfully.
  20000104.000000 :: ANALYSIS :: xinc.tile5 :: TEST was complete successfully.
  20000104.000000 :: ANALYSIS :: xinc.tile6 :: TEST was complete successfully.
  20000104.000000 :: ANALYSIS :: HofX :: TEST was complete successfully.
  20000104.000000 :: POST_ANAL :: ufs_land_restart.anal.tile1 :: TEST was complete successfully.
  20000104.000000 :: POST_ANAL :: ufs_land_restart.anal.tile2 :: TEST was complete successfully.
  20000104.000000 :: POST_ANAL :: ufs_land_restart.anal.tile3 :: TEST was complete successfully.
  20000104.000000 :: POST_ANAL :: ufs_land_restart.anal.tile4 :: TEST was complete successfully.
  20000104.000000 :: POST_ANAL :: ufs_land_restart.anal.tile5 :: TEST was complete successfully.
  20000104.000000 :: POST_ANAL :: ufs_land_restart.anal.tile6 :: TEST was complete successfully.
  20000104.000000 :: FORECAST :: ufs_land_restart.tile1 :: TEST was complete successfully.
  20000104.000000 :: FORECAST :: ufs_land_restart.tile2 :: TEST was complete successfully.
  20000104.000000 :: FORECAST :: ufs_land_restart.tile3 :: TEST was complete successfully.
  20000104.000000 :: FORECAST :: ufs_land_restart.tile4 :: TEST was complete successfully.
  20000104.000000 :: FORECAST :: ufs_land_restart.tile5 :: TEST was complete successfully.
  20000104.000000 :: FORECAST :: ufs_land_restart.tile6 :: TEST was complete successfully.