DavidHuber-NOAA commented 2 months ago

Description

Adds 3 hot fixes:

METplus v9.1.3 has a bug in it that sometimes attempts to create multiple copies of the same directory when running in parallel, causing a Python error and downstream problems. This PR makes METplus run in serial mode, preventing such issues.
Corrects the name of the atmos_prod, ocean_prod, and ice_prod tasks in workflow/rocoto/tasks.py (was accidentally changed to e.g. atmosprod)

Type of change

Bug fix (fixes something broken)

Change characteristics

Is this a breaking change (a change in existing functionality)? YES (For experiments running more than ~20 cycles, the gfsmetpg2o job may time out.)
Does this change require a documentation update? NO

How has this been tested?

CI unit tests for the atmos_prod fix, will run CI.

Checklist

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] My changes generate no new warnings
[ ] New and existing tests pass with my changes

emcbot commented 2 months ago


CI Update on Wcoss2 at 08/02/24 01:16:14 PM
============================================
Cloning and Building global-workflow PR: 2804
with PID: 191012 on host: clogin03

emcbot commented 2 months ago

Automated global-workflow Testing Results:


Machine: Wcoss2
Start: Fri Aug  2 13:21:41 UTC 2024 on clogin03
---------------------------------------------------
Build: Completed at 08/02/24 01:59:25 PM
Case setup: Completed for experiment C48_ATM_b738c51d
Case setup: Skipped for experiment C48mx500_3DVarAOWCDA_b738c51d
Case setup: Skipped for experiment C48_S2SWA_gefs_b738c51d
Case setup: Completed for experiment C48_S2SW_b738c51d
Case setup: Completed for experiment C96_atm3DVar_extended_b738c51d
Case setup: Skipped for experiment C96_atm3DVar_b738c51d
Case setup: Completed for experiment C96_atmaerosnowDA_b738c51d
Case setup: Completed for experiment C96C48_hybatmDA_b738c51d
Case setup: Completed for experiment C96C48_ufs_hybatmDA_b738c51d

emcbot commented 2 months ago

Experiment C48_ATM FAILED on Hera with error logs:

/scratch1/NCEPDEV/global/CI/2804/RUNTESTS/COMROOT/C48_ATM_b738c51d/logs/2021032312/gfsmetpg2o1.log

Follow link here to view the contents of the above file(s): (link)

emcbot commented 2 months ago

Experiment C48_ATM FAILED on Hera in /scratch1/NCEPDEV/global/CI/2804/RUNTESTS/EXPDIR/C48_ATM_b738c51d

DavidHuber-NOAA commented 2 months ago

@malloryprow Could you add the gdas/nam prepbufr files for 2021032300 through 2021033000 to Hera (under /scratch1/NCEPDEV/global/Mallory.Row/prepbufr/) ~and Orion/Hercules (under /work/noaa/rstprod/verif/prepbufr)~?

EDIT: Orion/Hercules actually already has this data.

emcbot commented 2 months ago

Experiment C48_S2SW FAILED on Hera with error logs:

/scratch1/NCEPDEV/global/CI/2804/RUNTESTS/COMROOT/C48_S2SW_b738c51d/logs/2021032312/gfsmetpg2o1.log

Follow link here to view the contents of the above file(s): (link)

emcbot commented 2 months ago

Experiment C48_S2SW FAILED on Hera in /scratch1/NCEPDEV/global/CI/2804/RUNTESTS/EXPDIR/C48_S2SW_b738c51d

emcbot commented 2 months ago

CI Failed Hera at
Built and ran in directory /scratch1/NCEPDEV/global/CI/2804



Experiment C48mx500_3DVarAOWCDA_b738c51d Completed 2 Cycles: *SUCCESS* at Fri Aug  2 15:34:00 UTC 2024
Experiment C48_ATM_b738c51d Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Fri Aug  2 15:40:04 UTC 2024
Experiment C48_ATM_b738c51d Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/2804/RUNTESTS/COMROOT/C48_ATM_b738c51d/logs/2021032312/gfsmetpg2o1.log
Experiment C96_atm3DVar_b738c51d Completed 3 Cycles: *SUCCESS* at Fri Aug  2 16:47:02 UTC 2024
Experiment C96C48_hybatmDA_b738c51d Completed 3 Cycles: *SUCCESS* at Fri Aug  2 16:47:45 UTC 2024
Experiment C48_S2SWA_gefs_b738c51d Completed 1 Cycles: *SUCCESS* at Fri Aug  2 16:47:47 UTC 2024
Experiment C48_S2SW_b738c51d Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Fri Aug  2 17:29:48 UTC 2024
Experiment C48_S2SW_b738c51d Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/2804/RUNTESTS/COMROOT/C48_S2SW_b738c51d/logs/2021032312/gfsmetpg2o1.log
Experiment C96_atmaerosnowDA_b738c51d Completed 3 Cycles: *SUCCESS* at Fri Aug  2 17:36:01 UTC 2024

emcbot commented 2 months ago

Experiment C96_atm3DVar_extended_b738c51d FAIL on Wcoss2 at 08/02/24 05:48:47 PM

Error logs:

/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2804/RUNTESTS/COMROOT/C96_atm3DVar_extended_b738c51d/logs/2021122100/gfsgempakgrb2spec_f000.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2804/RUNTESTS/COMROOT/C96_atm3DVar_extended_b738c51d/logs/2021122100/gfsgempakgrb2spec_f003.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2804/RUNTESTS/COMROOT/C96_atm3DVar_extended_b738c51d/logs/2021122100/gfsgempakgrb2spec_f006.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2804/RUNTESTS/COMROOT/C96_atm3DVar_extended_b738c51d/logs/2021122100/gfsgempakgrb2spec_f009.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2804/RUNTESTS/COMROOT/C96_atm3DVar_extended_b738c51d/logs/2021122100/gfsgempakgrb2spec_f012.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2804/RUNTESTS/COMROOT/C96_atm3DVar_extended_b738c51d/logs/2021122100/gfsgempakgrb2spec_f015.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2804/RUNTESTS/COMROOT/C96_atm3DVar_extended_b738c51d/logs/2021122100/gfsgempakgrb2spec_f018.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2804/RUNTESTS/COMROOT/C96_atm3DVar_extended_b738c51d/logs/2021122100/gfsgempakgrb2spec_f021.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2804/RUNTESTS/COMROOT/C96_atm3DVar_extended_b738c51d/logs/2021122100/gfsgempakgrb2spec_f024.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2804/RUNTESTS/COMROOT/C96_atm3DVar_extended_b738c51d/logs/2021122100/gfsgempakgrb2spec_f027.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2804/RUNTESTS/COMROOT/C96_atm3DVar_extended_b738c51d/logs/2021122100/gfsgempakgrb2spec_f030.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2804/RUNTESTS/COMROOT/C96_atm3DVar_extended_b738c51d/logs/2021122100/gfsgempakgrb2spec_f033.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2804/RUNTESTS/COMROOT/C96_atm3DVar_extended_b738c51d/logs/2021122100/gfsgempakgrb2spec_f036.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2804/RUNTESTS/COMROOT/C96_atm3DVar_extended_b738c51d/logs/2021122100/gfsgempakgrb2spec_f039.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2804/RUNTESTS/COMROOT/C96_atm3DVar_extended_b738c51d/logs/2021122100/gfsgempakgrb2spec_f042.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2804/RUNTESTS/COMROOT/C96_atm3DVar_extended_b738c51d/logs/2021122100/gfsgempakgrb2spec_f045.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2804/RUNTESTS/COMROOT/C96_atm3DVar_extended_b738c51d/logs/2021122100/gfsgempakgrb2spec_f048.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2804/RUNTESTS/COMROOT/C96_atm3DVar_extended_b738c51d/logs/2021122100/gfsgempakgrb2spec_f051.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2804/RUNTESTS/COMROOT/C96_atm3DVar_extended_b738c51d/logs/2021122100/gfsgempakgrb2spec_f054.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2804/RUNTESTS/COMROOT/C96_atm3DVar_extended_b738c51d/logs/2021122100/gfsgempakgrb2spec_f057.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2804/RUNTESTS/COMROOT/C96_atm3DVar_extended_b738c51d/logs/2021122100/gfsgempakgrb2spec_f060.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2804/RUNTESTS/COMROOT/C96_atm3DVar_extended_b738c51d/logs/2021122100/gfsgempakmeta.log

Follow link here to view the contents of the above file(s): (link)

DavidHuber-NOAA commented 2 months ago

@aerorahul @WalterKolczynski-NOAA The failures on WCOSS2 are due to a couple of JJob scripts that do not source config.resources. These will be fixed in #2753. The failures in gfsmetpg2o1 on Hera are due to missing prepbufr input files and will be uploaded soon.

I believe this PR is ready to merge.

DavidHuber-NOAA commented 2 months ago

The prepbufr files have been pushed to Hera. Thanks @malloryprow!!

NOAA-EMC / global-workflow

Run METplus serially and correct the name of prod tasks #2804

Description

Type of change

Change characteristics

How has this been tested?

Checklist