CoryMartin-NOAA commented 1 week ago

Description

This PR adds DO_TEST_MODE, which can be used for other things in the future but for now sets the GSI to run just 5 iterations per outer loop to reduce runtime for CI testing.

Resolves #3114

Type of change

[ ] Bug fix (fixes something broken)
[ ] New feature (adds functionality)
[x] Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? MAYBE
Does this change require an update to any of the following submodules? NO

How has this been tested?

Clone and build on Hera
Cycled test on Hera

Checklist

[x] Any dependent changes have been merged and published
[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have documented my code, including function, input, and output descriptions
[x] My changes generate no new warnings
[x] New and existing tests pass with my changes
[x] This change is covered by an existing CI test or a new one has been added
[x] Any new scripts have been added to the .github/CODEOWNERS file with owners
[x] I have made corresponding changes to the system documentation if necessary

emcbot commented 1 week ago

Experiment C48_S2SW FAILED on Hera in Build# 2 with error logs:

/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C48_S2SW_0bb9078e/logs/2021032312/gfs_stage_ic.log

Follow link here to view the contents of the above file(s): (link)

emcbot commented 1 week ago

Experiment C48_S2SW FAILED on Hera in Build# 2 in /scratch1/NCEPDEV/global/CI/3115/RUNTESTS/EXPDIR/C48_S2SW_0bb9078e

JessicaMeixner-NOAA commented 1 week ago

@WalterKolczynski-NOAA I was pointed to this failure offline because it was about a wave IC issue:

SError: unable to copy /scratch1/NCEPDEV/global/glopara/data/ICSDIR/C48mx500/20240610/gfs.20210323/06/model/wave/restart/20210323.120000.restart.ww3 to /scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C48_S2SW_0bb9078e/gfs.20210323/06//model/wave/restart/20210323.120000.restart.ww3

I'm concerned that changes in staging of wave ICs has messed up existing regression tests in prep for https://github.com/NOAA-EMC/global-workflow/pull/3112 removing the grid id perhaps is causing an issue because we have many wave gridids that are used in various places and what was copied to ww3? And where? I'll continue this conversation in 3112. I don't think we want to break other tests before 3112 is merged do we? And perahps we need to carefully go through and make sure ICs are properly placed for various wave grids for various CI tests. I'm unsure about this now.

emcbot commented 1 week ago

Experiment C48_S2SWA_gefs FAILED on Hera in Build# 2 with error logs:

/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C48_S2SWA_gefs_0bb9078e/logs/2021032312/gefs_stage_ic.log

Follow link here to view the contents of the above file(s): (link)

emcbot commented 1 week ago

Experiment C48_S2SWA_gefs FAILED on Hera in Build# 2 in /scratch1/NCEPDEV/global/CI/3115/RUNTESTS/EXPDIR/C48_S2SWA_gefs_0bb9078e

WalterKolczynski-NOAA commented 1 week ago

@WalterKolczynski-NOAA I was pointed to this failure offline because it was about a wave IC issue:

SError: unable to copy /scratch1/NCEPDEV/global/glopara/data/ICSDIR/C48mx500/20240610/gfs.20210323/06/model/wave/restart/20210323.120000.restart.ww3 to /scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C48_S2SW_0bb9078e/gfs.20210323/06//model/wave/restart/20210323.120000.restart.ww3

I'm concerned that changes in staging of wave ICs has messed up existing regression tests in prep for #3112 removing the grid id perhaps is causing an issue because we have many wave gridids that are used in various places and what was copied to ww3? And where? I'll continue this conversation in 3112. I don't think we want to break other tests before 3112 is merged do we? And perahps we need to carefully go through and make sure ICs are properly placed for various wave grids for various CI tests. I'm unsure about this now.

A few symlinks got broken somehow. Breaking old tests was not intentional. The links are restored now.

emcbot commented 1 week ago

Experiment C96_S2SWA_gefs_replay_ics FAILED on Hera in Build# 2 in /scratch1/NCEPDEV/global/CI/3115/RUNTESTS/EXPDIR/C96_S2SWA_gefs_replay_ics_0bb9078e

emcbot commented 1 week ago

Experiment C96_atm3DVar FAILED on Hera in Build# 2 in /scratch1/NCEPDEV/global/CI/3115/RUNTESTS/EXPDIR/C96_atm3DVar_0bb9078e

emcbot commented 1 week ago

Experiment C96C48_ufs_hybatmDA FAILED on Hera in Build# 2 in /scratch1/NCEPDEV/global/CI/3115/RUNTESTS/EXPDIR/C96C48_ufs_hybatmDA_0bb9078e

emcbot commented 1 week ago

Experiment C96C48_hybatmaerosnowDA FAILED on Hera in Build# 2 in /scratch1/NCEPDEV/global/CI/3115/RUNTESTS/EXPDIR/C96C48_hybatmaerosnowDA_0bb9078e

emcbot commented 1 week ago

Experiment C96C48_hybatmDA FAILED on Hera in Build# 2 in /scratch1/NCEPDEV/global/CI/3115/RUNTESTS/EXPDIR/C96C48_hybatmDA_0bb9078e

emcbot commented 1 week ago

CI Passed on Hercules in Build# 1
Built and ran in directory /work2/noaa/global/CI/HERCULES/3115



Experiment C48_ATM_0bb9078e Completed 2 Cycles: *SUCCESS* at Fri Nov 22 09:22:22 CST 2024
Experiment C96_S2SWA_gefs_replay_ics_0bb9078e Completed 1 Cycles: *SUCCESS* at Fri Nov 22 09:22:28 CST 2024
Experiment C48_S2SW_0bb9078e Completed 2 Cycles: *SUCCESS* at Fri Nov 22 10:47:28 CST 2024
Experiment C96_atm3DVar_0bb9078e Completed 3 Cycles: *SUCCESS* at Fri Nov 22 10:47:30 CST 2024
Experiment C96C48_hybatmDA_0bb9078e Completed 3 Cycles: *SUCCESS* at Fri Nov 22 10:47:38 CST 2024
Experiment C48_S2SWA_gefs_0bb9078e Completed 1 Cycles: *SUCCESS* at Fri Nov 22 11:37:12 CST 2024

emcbot commented 1 week ago

Experiment C96_atm3DVar FAILED on Hera in Build# 3 with error logs:

/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f000.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f003.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f006.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f009.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f012.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f015.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f018.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f021.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f024.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f027.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f030.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f033.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f036.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f039.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f042.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f045.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f048.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f051.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f054.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f057.log

Follow link here to view the contents of the above file(s): (link)

emcbot commented 1 week ago

Experiment C96_atm3DVar FAILED on Hera in Build# 3 in /scratch1/NCEPDEV/global/CI/3115/RUNTESTS/EXPDIR/C96_atm3DVar_0bb9078e

emcbot commented 1 week ago

Experiment C48_S2SWA_gefs FAILED on Hera in Build# 3 in /scratch1/NCEPDEV/global/CI/3115/RUNTESTS/EXPDIR/C48_S2SWA_gefs_0bb9078e

emcbot commented 1 week ago

Experiment C48_S2SW FAILED on Hera in Build# 3 in /scratch1/NCEPDEV/global/CI/3115/RUNTESTS/EXPDIR/C48_S2SW_0bb9078e

emcbot commented 1 week ago

Experiment C96C48_hybatmaerosnowDA FAILED on Hera in Build# 3 in /scratch1/NCEPDEV/global/CI/3115/RUNTESTS/EXPDIR/C96C48_hybatmaerosnowDA_0bb9078e

emcbot commented 1 week ago

Experiment C96C48_ufs_hybatmDA FAILED on Hera in Build# 3 in /scratch1/NCEPDEV/global/CI/3115/RUNTESTS/EXPDIR/C96C48_ufs_hybatmDA_0bb9078e

emcbot commented 1 week ago

Experiment C96C48_hybatmDA FAILED on Hera in Build# 3 in /scratch1/NCEPDEV/global/CI/3115/RUNTESTS/EXPDIR/C96C48_hybatmDA_0bb9078e

emcbot commented 1 week ago

CI Failed on Hera in Build# 3
Built and ran in directory /scratch1/NCEPDEV/global/CI/3115



Experiment C96_S2SWA_gefs_replay_ics_0bb9078e Completed 1 Cycles: *SUCCESS* at Fri Nov 22 19:26:19 UTC 2024
Experiment C96_atm3DVar_0bb9078e Terminated with 0
FAIL
FAIL tasks failed and 20 dead at Fri Nov 22 19:56:33 UTC 2024
Experiment C96_atm3DVar_0bb9078e Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f000.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f003.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f006.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f009.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f012.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f015.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f018.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f021.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f024.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f027.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f030.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f033.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f036.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f039.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f042.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f045.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f048.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f051.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f054.log
/scratch1/NCEPDEV/global/CI/3115/RUNTESTS/COMROOT/C96_atm3DVar_0bb9078e/logs/2021122100/gfs_atmos_prod_f057.log
Experiment C48mx500_3DVarAOWCDA_0bb9078e Completed 2 Cycles: *SUCCESS* at Fri Nov 22 20:02:37 UTC 2024
Experiment C48_ATM_0bb9078e Completed 2 Cycles: *SUCCESS* at Fri Nov 22 20:02:38 UTC 2024

emcbot commented 1 week ago

Checkout Failed on Hera in Build# 5: null

emcbot commented 1 week ago

Checkout Failed on Hera in Build# 5: null

emcbot commented 6 days ago

CI Passed on Hera in Build# 7
Built and ran in directory /scratch1/NCEPDEV/global/CI/3115



Experiment C96_S2SWA_gefs_replay_ics_baccdedd Completed 1 Cycles: *SUCCESS* at Sat Nov 23 13:15:45 UTC 2024
Experiment C48mx500_3DVarAOWCDA_baccdedd Completed 2 Cycles: *SUCCESS* at Sat Nov 23 13:27:55 UTC 2024
Experiment C48_ATM_baccdedd Completed 2 Cycles: *SUCCESS* at Sat Nov 23 13:33:59 UTC 2024
Experiment C96C48_hybatmaerosnowDA_baccdedd Completed 3 Cycles: *SUCCESS* at Sat Nov 23 14:47:18 UTC 2024
Experiment C96C48_hybatmDA_baccdedd Completed 3 Cycles: *SUCCESS* at Sat Nov 23 14:47:19 UTC 2024
Experiment C48_S2SWA_gefs_baccdedd Completed 1 Cycles: *SUCCESS* at Sat Nov 23 14:48:19 UTC 2024
Experiment C96_atm3DVar_baccdedd Completed 3 Cycles: *SUCCESS* at Sat Nov 23 14:53:11 UTC 2024
Experiment C48_S2SW_baccdedd Completed 2 Cycles: *SUCCESS* at Sat Nov 23 15:23:38 UTC 2024
Experiment C96C48_ufs_hybatmDA_baccdedd Completed 3 Cycles: *SUCCESS* at Sat Nov 23 15:30:11 UTC 2024

NOAA-EMC / global-workflow

Speed up GSI analysis jobs in CI testing #3115

Description

Type of change

Change characteristics

How has this been tested?

Checklist