NOAA-EMC / global-workflow

Global Superstructure/Workflow supporting the Global Forecast System (GFS)

https://global-workflow.readthedocs.io/en/latest

GNU Lesser General Public License v3.0

75 stars 170 forks source link

Use stochastic restart patterns on rerun #3077

Open WalterKolczynski-NOAA opened 3 weeks ago

WalterKolczynski-NOAA commented 3 weeks ago

Description

The stochastic pattern restart files were not being copied into the input directory when restarting the model after a segment/failure. These files are now copied in and the stochini flag set to .true. in the namelist on a rerun.

The files are NOT copied in for non-rerun warm starts.

Also removes the restriction that stochastic physics cannot be run on member 0, as this is desired down the line. Additional settings are added to the fcst and efcs configs to retain the current behavior.

A bug was also discovered and corrected during this work. The stage, forecast, and archive job all assumed that ca_data tile files are always present, but these are only created when cellular automata is on. Now ca_data files are handled if CA is on. To implement this change, the DO_CA setting had to be moved from the forecast configs to base so it is available to the stage_ic and archive jobs.

Resolves #2937

Type of change

[x] Bug fix (fixes something broken)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO
Does this change require an update to any of the following submodules? NO

How has this been tested?

Multi-segment test on Hercules

Checklist

[x] Any dependent changes have been merged and published
[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have documented my code, including function, input, and output descriptions
[x] My changes generate no new warnings
[x] New and existing tests pass with my changes
[x] This change is covered by an existing CI test or a new one has been added
[x] Any new scripts have been added to the .github/CODEOWNERS file with owners
[x] I have made corresponding changes to the system documentation if necessary

emcbot commented 2 weeks ago

Experiment C96_atm3DVar FAILED on Hera in Build# 1 in /scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C96_atm3DVar_d9209edc

emcbot commented 2 weeks ago

Experiment C48mx500_3DVarAOWCDA FAILED on Hera in Build# 1 with error logs:

/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_d9209edc/logs/2021032412/gdas_fcst_seg0.log

Follow link here to view the contents of the above file(s): (link)

emcbot commented 2 weeks ago

Experiment C48mx500_3DVarAOWCDA FAILED on Hera in Build# 1 in /scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C48mx500_3DVarAOWCDA_d9209edc

emcbot commented 2 weeks ago

Experiment C96C48_ufs_hybatmDA FAILED on Hera in Build# 1 with error logs:

/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96C48_ufs_hybatmDA_d9209edc/logs/2024022318/gdas_fcst_seg0.log

Follow link here to view the contents of the above file(s): (link)

emcbot commented 2 weeks ago

Experiment C96C48_hybatmDA FAILED on Hera in Build# 1 with error logs:

/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96C48_hybatmDA_d9209edc/logs/2021122018/gdas_fcst_seg0.log

Follow link here to view the contents of the above file(s): (link)

emcbot commented 2 weeks ago

Experiment C96C48_ufs_hybatmDA FAILED on Hera in Build# 1 in /scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C96C48_ufs_hybatmDA_d9209edc

emcbot commented 2 weeks ago

Experiment C96C48_hybatmaerosnowDA FAILED on Hera in Build# 1 with error logs:

/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96C48_hybatmaerosnowDA_d9209edc/logs/2021122012/gdas_fcst_seg0.log

Follow link here to view the contents of the above file(s): (link)

emcbot commented 2 weeks ago

Experiment C96C48_hybatmDA FAILED on Hera in Build# 1 in /scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C96C48_hybatmDA_d9209edc

emcbot commented 2 weeks ago

Experiment C96C48_hybatmaerosnowDA FAILED on Hera in Build# 1 in /scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C96C48_hybatmaerosnowDA_d9209edc

emcbot commented 2 weeks ago

Experiment C48_S2SWA_gefs FAILED on Hera in Build# 1 with error logs:

/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_d9209edc/logs/2021032312/gefs_fcst_mem000_seg1.log
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_d9209edc/logs/2021032312/gefs_fcst_mem001_seg1.log
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_d9209edc/logs/2021032312/gefs_fcst_mem002_seg1.log

Follow link here to view the contents of the above file(s): (link)

emcbot commented 2 weeks ago

Experiment C48_S2SWA_gefs FAILED on Hera in Build# 1 in /scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C48_S2SWA_gefs_d9209edc

emcbot commented 2 weeks ago

CI Failed on Hera in Build# 1
Built and ran in directory /scratch1/NCEPDEV/global/CI/3077



Experiment C48mx500_3DVarAOWCDA_d9209edc Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Nov 13 11:45:45 UTC 2024
Experiment C48mx500_3DVarAOWCDA_d9209edc Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_d9209edc/logs/2021032412/gdas_fcst_seg0.log
Experiment C96_atm3DVar_d9209edc Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Nov 13 11:45:50 UTC 2024
Experiment C96_atm3DVar_d9209edc Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96_atm3DVar_d9209edc/logs/2021122018/gdas_fcst_seg0.log
Experiment C96C48_hybatmDA_d9209edc Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Nov 13 11:51:56 UTC 2024
Experiment C96C48_hybatmDA_d9209edc Terminated: *FAIL*
Experiment C96C48_ufs_hybatmDA_d9209edc Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Nov 13 11:51:57 UTC 2024
Experiment C96C48_ufs_hybatmDA_d9209edc Terminated: *FAIL*
Experiment C96C48_hybatmaerosnowDA_d9209edc Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Nov 13 11:51:59 UTC 2024
Experiment C96C48_hybatmaerosnowDA_d9209edc Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96C48_hybatmDA_d9209edc/logs/2021122018/gdas_fcst_seg0.log
Error logs:
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96C48_ufs_hybatmDA_d9209edc/logs/2024022318/gdas_fcst_seg0.log
Error logs:
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96C48_hybatmaerosnowDA_d9209edc/logs/2021122012/gdas_fcst_seg0.log
Experiment C48_S2SWA_gefs_d9209edc Terminated with 0
FAIL
FAIL tasks failed and 3 dead at Wed Nov 13 15:37:41 UTC 2024
Experiment C48_S2SWA_gefs_d9209edc Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_d9209edc/logs/2021032312/gefs_fcst_mem000_seg1.log
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_d9209edc/logs/2021032312/gefs_fcst_mem001_seg1.log
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_d9209edc/logs/2021032312/gefs_fcst_mem002_seg1.log
Experiment C96_S2SWA_gefs_replay_ics_d9209edc Completed 1 Cycles: *SUCCESS* at Wed Nov 13 17:25:52 UTC 2024
Experiment C48_ATM_d9209edc Completed 2 Cycles: *SUCCESS* at Wed Nov 13 18:45:20 UTC 2024
Experiment C48_S2SW_d9209edc Completed 2 Cycles: *SUCCESS* at Wed Nov 13 23:13:39 UTC 2024

emcbot commented 4 days ago

Experiment C48_S2SWA_gefs FAILED on Hercules in Build# 4 with error logs:

/work2/noaa/global/CI/HERCULES/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_20705b04/logs/2021032312/gefs_fcst_mem001_seg1.log
/work2/noaa/global/CI/HERCULES/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_20705b04/logs/2021032312/gefs_fcst_mem002_seg1.log

Follow link here to view the contents of the above file(s): (link)

emcbot commented 4 days ago

Experiment C48_S2SWA_gefs FAILED on Hercules in Build# 4 in /work2/noaa/global/CI/HERCULES/3077/RUNTESTS/EXPDIR/C48_S2SWA_gefs_20705b04

emcbot commented 4 days ago

CI Failed on Hercules in Build# 4
Built and ran in directory /work2/noaa/global/CI/HERCULES/3077



Experiment C48_S2SWA_gefs_20705b04 Terminated with 0
FAIL
FAIL tasks failed and 2 dead at Mon Nov 25 12:42:50 CST 2024
Experiment C48_S2SWA_gefs_20705b04 Terminated: *FAIL*
Error logs:
/work2/noaa/global/CI/HERCULES/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_20705b04/logs/2021032312/gefs_fcst_mem001_seg1.log
/work2/noaa/global/CI/HERCULES/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_20705b04/logs/2021032312/gefs_fcst_mem002_seg1.log
Experiment C48_ATM_20705b04 Completed 2 Cycles: *SUCCESS* at Mon Nov 25 13:12:38 CST 2024
Experiment C96_S2SWA_gefs_replay_ics_20705b04 Completed 1 Cycles: *SUCCESS* at Mon Nov 25 13:12:41 CST 2024
Experiment C48_S2SW_20705b04 Completed 2 Cycles: *SUCCESS* at Mon Nov 25 14:13:10 CST 2024
Experiment C96_atm3DVar_20705b04 Completed 3 Cycles: *SUCCESS* at Mon Nov 25 14:19:14 CST 2024
Experiment C96C48_hybatmDA_20705b04 Completed 3 Cycles: *SUCCESS* at Mon Nov 25 14:19:35 CST 2024

WalterKolczynski-NOAA commented 3 days ago

This works for me manually. I'm going to try again and inspect it as it runs.

emcbot commented 3 days ago

Experiment C48_S2SWA_gefs FAILED on Hercules in Build# 5 with error logs:

/work2/noaa/global/CI/HERCULES/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_dbc81126/logs/2021032312/gefs_fcst_mem001_seg1.log
/work2/noaa/global/CI/HERCULES/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_dbc81126/logs/2021032312/gefs_fcst_mem002_seg1.log

Follow link here to view the contents of the above file(s): (link)

emcbot commented 3 days ago

Experiment C48_S2SWA_gefs FAILED on Hercules in Build# 5 in /work2/noaa/global/CI/HERCULES/3077/RUNTESTS/EXPDIR/C48_S2SWA_gefs_dbc81126

emcbot commented 3 days ago

Experiment C48_S2SW FAILED on Hercules in Build# 5 in /work2/noaa/global/CI/HERCULES/3077/RUNTESTS/EXPDIR/C48_S2SW_dbc81126

emcbot commented 3 days ago

Experiment C96_atm3DVar FAILED on Hercules in Build# 5 in /work2/noaa/global/CI/HERCULES/3077/RUNTESTS/EXPDIR/C96_atm3DVar_dbc81126

emcbot commented 3 days ago

Experiment C48_ATM FAILED on Hercules in Build# 5 in /work2/noaa/global/CI/HERCULES/3077/RUNTESTS/EXPDIR/C48_ATM_dbc81126

emcbot commented 3 days ago

Experiment C96_S2SWA_gefs_replay_ics FAILED on Hercules in Build# 5 in /work2/noaa/global/CI/HERCULES/3077/RUNTESTS/EXPDIR/C96_S2SWA_gefs_replay_ics_dbc81126

emcbot commented 3 days ago

Experiment C96C48_hybatmDA FAILED on Hercules in Build# 5 in /work2/noaa/global/CI/HERCULES/3077/RUNTESTS/EXPDIR/C96C48_hybatmDA_dbc81126

emcbot commented 3 days ago

CI Failed on Hercules in Build# 5
Built and ran in directory /work2/noaa/global/CI/HERCULES/3077



Experiment C48_S2SWA_gefs_dbc81126 Terminated with 0
FAIL
FAIL tasks failed and 2 dead at Tue Nov 26 13:29:05 CST 2024
Experiment C48_S2SWA_gefs_dbc81126 Terminated: *FAIL*
Error logs:
/work2/noaa/global/CI/HERCULES/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_dbc81126/logs/2021032312/gefs_fcst_mem001_seg1.log
/work2/noaa/global/CI/HERCULES/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_dbc81126/logs/2021032312/gefs_fcst_mem002_seg1.log

emcbot commented 3 days ago

Experiment C48_S2SWA_gefs FAILED on Hercules in Build# 6 with error logs:

/work2/noaa/global/CI/HERCULES/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_c38dbcfd/logs/2021032312/gefs_fcst_mem001_seg1.log
/work2/noaa/global/CI/HERCULES/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_c38dbcfd/logs/2021032312/gefs_fcst_mem002_seg1.log

Follow link here to view the contents of the above file(s): (link)

emcbot commented 3 days ago

Experiment C48_S2SWA_gefs FAILED on Hercules in Build# 6 in /work2/noaa/global/CI/HERCULES/3077/RUNTESTS/EXPDIR/C48_S2SWA_gefs_c38dbcfd

emcbot commented 3 days ago

CI Failed on Hercules in Build# 6
Built and ran in directory /work2/noaa/global/CI/HERCULES/3077



Experiment C48_S2SWA_gefs_c38dbcfd Terminated with 0
FAIL
FAIL tasks failed and 2 dead at Tue Nov 26 15:07:26 CST 2024
Experiment C48_S2SWA_gefs_c38dbcfd Terminated: *FAIL*
Error logs:
/work2/noaa/global/CI/HERCULES/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_c38dbcfd/logs/2021032312/gefs_fcst_mem001_seg1.log
/work2/noaa/global/CI/HERCULES/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_c38dbcfd/logs/2021032312/gefs_fcst_mem002_seg1.log
Experiment C48_ATM_c38dbcfd Completed 2 Cycles: *SUCCESS* at Tue Nov 26 15:55:34 CST 2024
Experiment C96_S2SWA_gefs_replay_ics_c38dbcfd Completed 1 Cycles: *SUCCESS* at Tue Nov 26 16:13:51 CST 2024
Experiment C96_atm3DVar_c38dbcfd Completed 3 Cycles: *SUCCESS* at Tue Nov 26 17:02:12 CST 2024
Experiment C96C48_hybatmDA_c38dbcfd Completed 3 Cycles: *SUCCESS* at Tue Nov 26 17:02:24 CST 2024
Experiment C48_S2SW_c38dbcfd Completed 2 Cycles: *SUCCESS* at Tue Nov 26 17:38:29 CST 2024