search

NOAA-EMC / GDASApp

Global Data Assimilation System Application
GNU Lesser General Public License v2.1
14 stars 28 forks source link

replace model_data with model #1202

Closed RussTreadon-NOAA closed 6 days ago

RussTreadon-NOAA commented 6 days ago

NCO requires model_data be replaced with model in the path to model data. This PR satisfies NCO's requirement.

Resolves #1198

RussTreadon-NOAA commented 6 days ago

Completion of this PR requires model_data be replaced with model in lowres data paths in GDASAPP_TESTDATA on WCOSS2, Hera, Hercules, and Orion. Specifically the following needs to be done

The above has been done on

Currently modulefiles/GDAS/hera.intel.lua sets

setenv("GDASAPP_TESTDATA","/scratch1/NCEPDEV/da/Cory.R.Martin/CI/GDASApp/data")

A modified copy of hera.intel.lua was committed to feature/model at adb1b6d. The modified hera modulefile in feature/model points GDASAPP_TESTDATA at role.jedipara

@CoryMartin-NOAA and @guillaumevernieres , I see role-da on Orion/Hercules. I can access this account. However, is this the role account we want to use on Orion/Hercules for GDASAPP data? Do we want a new EMC-DA role account on Orion/Hercules?

CoryMartin-NOAA commented 6 days ago

@RussTreadon-NOAA I believe that is the role account we have access to for this purpose

RussTreadon-NOAA commented 6 days ago

Hera ctests Install feature/model inside working copy of g-w PR #2700. Run test_gdasapp. All tests pass

Test project /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/pr2700/sorc/gdas.cd/build
      Start 1488: test_gdasapp_util_coding_norms
 1/48 Test #1488: test_gdasapp_util_coding_norms ........................   Passed    2.72 sec
      Start 1489: test_gdasapp_util_ioda_example
 2/48 Test #1489: test_gdasapp_util_ioda_example ........................   Passed    2.39 sec
...
      Start 1869: test_gdasapp_aero_gen_3dvar_yaml
48/48 Test #1869: test_gdasapp_aero_gen_3dvar_yaml ......................   Passed    1.25 sec

100% tests passed, 0 tests failed out of 48

Label Time Summary:
gdas-utils    =  17.73 sec*proc (11 tests)
script        =  17.73 sec*proc (11 tests)

Total Test time (real) = 3529.84 sec

Orion ctests Install feature/model inside working copy of g-w PR #2700. Run test_gdasapp. All tests pass

Test project /work2/noaa/da/rtreadon/git/global-workflow/rename_atm/sorc/gdas.cd/build
      Start 1489: test_gdasapp_util_coding_norms
 1/48 Test #1489: test_gdasapp_util_coding_norms ........................   Passed    6.68 sec
      Start 1490: test_gdasapp_util_ioda_example
 2/48 Test #1490: test_gdasapp_util_ioda_example ........................   Passed    2.51 sec
...
      Start 1870: test_gdasapp_aero_gen_3dvar_yaml
48/48 Test #1870: test_gdasapp_aero_gen_3dvar_yaml ......................   Passed    0.54 sec

100% tests passed, 0 tests failed out of 48

Label Time Summary:
gdas-utils    =  15.32 sec*proc (11 tests)
script        =  15.32 sec*proc (11 tests)

Total Test time (real) = 1884.31 sec

WCOSS2 (Dogwood) ctests Install feature/model inside working copy of g-w PR #2700. Run test_gdasapp. 20 our of 46 tests fail. This is a known problem. Several GDASApp ctests assume SLURM is available. WCOSS2 uses PBS. The tests which failed using feature/model are the same tests which failed in the past.

57% tests passed, 20 tests failed out of 46

Label Time Summary:
gdas-utils    =  10.47 sec*proc (9 tests)
script        =  10.47 sec*proc (9 tests)

Total Test time (real) = 358.00 sec

The following tests FAILED:
        1751 - test_gdasapp_fv3jedi_fv3inc (Not Run)
        1756 - test_gdasapp_soca_JGLOBAL_PREP_OCEAN_OBS (Failed)
        1757 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP (Failed)
        1758 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT (Failed)
        1759 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN (Failed)
        1760 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_ECEN (Failed)
        1761 - test_gdasapp_soca_copy_scratch (Failed)
        1762 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT (Failed)
        1763 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST (Failed)
        1764 - test_gdasapp_soca_socahybridweights (Failed)
        1765 - test_gdasapp_soca_incr_handler (Failed)
        1766 - test_gdasapp_soca_ens_handler (Failed)
        1769 - test_gdasapp_snow_apply_jediincr (Failed)
        1770 - test_gdasapp_snow_letkfoi_snowda (Failed)
        1776 - test_gdasapp_atm_jjob_var_run (Failed)
        1777 - test_gdasapp_atm_jjob_var_inc (Failed)
        1778 - test_gdasapp_atm_jjob_var_final (Failed)
        1780 - test_gdasapp_atm_jjob_ens_run (Failed)
        1781 - test_gdasapp_atm_jjob_ens_inc (Failed)
        1782 - test_gdasapp_atm_jjob_ens_final (Failed)
RussTreadon-NOAA commented 6 days ago

@CoryMartin-NOAA , I agree that the problem Yongzou reports is related to emcrzdm issues. However, doesn't the build first check if the tarfile is available from disk? If true, my renaming of the test directories on WCOSS2 (Dogwood) will cause problems for developers. The develop wcoss2.intel.lua has

setenv("CRTM_FIX","/lfs/h2/emc/da/noscrub/emc.da/GDASApp/fix/crtm/2.4.0")
setenv("GDASAPP_TESTDATA","/lfs/h2/emc/da/noscrub/emc.da/GDASApp/data")
setenv("GDASAPP_UNIT_TEST_DATA_PATH", "/lfs/h2/emc/da/noscrub/emc.da/GDASApp/data/test")

These paths have been changed to

setenv("CRTM_FIX","/lfs/h2/emc/da/noscrub/emc.da/GDASApp/fix/crtm/2.4.0")
setenv("GDASAPP_TESTDATA","/lfs/h2/emc/da/noscrub/emc.da/GDASApp/testdata")
setenv("GDASAPP_UNIT_TEST_DATA_PATH", "/lfs/h2/emc/da/noscrub/emc.da/GDASApp/unittestdata")

The Hera, Hercules, and Orion modulefiles in develop still point at your directories.

CoryMartin-NOAA commented 6 days ago

Good point @RussTreadon-NOAA

RussTreadon-NOAA commented 6 days ago

@CoryMartin-NOAA : If we are OK with this PR in it's current form, we should merge since some developers are building GDASApp on WCOSS2.