NOAA-EMC / global-workflow

Global Superstructure/Workflow supporting the Global Forecast System (GFS)

https://global-workflow.readthedocs.io/en/latest

GNU Lesser General Public License v3.0

75 stars 168 forks source link

Remove need for a ice_prod dependency check script #2809

Closed EricSinsky-NOAA closed 1 month ago

EricSinsky-NOAA commented 3 months ago

Description

The main purpose of this PR is to remove the need for an ice_prod dependency check script ush/check_ice_netcdf.sh. The original purpose of the ice_prod dependency check script is to check for special case dependencies where ( cyc + FHMIN ) % FHOUT_ICE )) =! 0 (more details on this issue can be found in issue #2674 ). A bugfix for these special cases is expected to come from a PR in the ufs-weather-model.

Resolves #2721 Refs #2721, #2674

Type of change

New feature (adds functionality)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO
Does this change require an update to any of the following submodules? NO

How has this been tested?

Clone, built and tested on WCOSS2.
In the WCOSS2 test, FHOUT_OCN and FHOUT_ICE was set to 24 and the GEFS CI case was successfully tested (202103232 12Z).
TODO: Perform test cases with FHOUT_OCN and FHOUT_ICE set to 6.

Checklist

[x] Any dependent changes have been merged and published
[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] My changes generate no new warnings
[x] New and existing tests pass with my changes
[ ] I have made corresponding changes to the documentation if necessary

EricSinsky-NOAA commented 2 months ago

Bugfix has been removed for special cases where ( cyc + FHMIN ) % FHOUT_ICE )) =! 0 in commit fcb9003f44892656a05ab7b82f58c1e0107f0753. This bug is expected to be fixed in ufs_weather_model PR #2437. The primary purpose of this PR is to remove ice_prod dependency check script and any temporary bugfixes associated with special cases where ( cyc + FHMIN ) % FHOUT_ICE )) =! 0. Testing GEFS replay and non-replay cases. This PR will be marked ready for review when these tests pass.

EricSinsky-NOAA commented 2 months ago

Marking this PR as ready for review. GEFS replay (FHOUT_GFS_ICE=24, cyc=00) and non-replay test cases (FHOUT_GFS_ICE=6, cyc=12) have passed.

emcbot commented 2 months ago

CI Passed on Hercules in Build# 1
Built and ran in directory /work2/noaa/stmp/CI/HERCULES/2809



Experiment C48_ATM_6c8548e3 Completed 1 Cycles: *SUCCESS* at Fri Sep 20 03:33:54 CDT 2024
Experiment C48_S2SW_6c8548e3 Completed 1 Cycles: *SUCCESS* at Fri Sep 20 04:28:17 CDT 2024
Experiment C96_atm3DVar_6c8548e3 Completed 3 Cycles: *SUCCESS* at Fri Sep 20 04:46:25 CDT 2024
Experiment C96C48_hybatmDA_6c8548e3 Completed 3 Cycles: *SUCCESS* at Fri Sep 20 04:46:39 CDT 2024
Experiment C48_S2SWA_gefs_6c8548e3 Completed 1 Cycles: *SUCCESS* at Fri Sep 20 05:11:26 CDT 2024

emcbot commented 2 months ago


CI Update on Wcoss2 at 09/20/24 03:10:11 PM
============================================
Cloning and Building global-workflow PR: 2809
with PID: 198381 on host: clogin03

emcbot commented 2 months ago

Automated global-workflow Testing Results:


Machine: Wcoss2
Start: Fri Sep 20 15:15:27 UTC 2024 on clogin03
---------------------------------------------------
Build: Completed at 09/20/24 03:52:50 PM
Case setup: Completed for experiment C48_ATM_6c8548e3
Case setup: Skipped for experiment C48mx500_3DVarAOWCDA_6c8548e3
Case setup: Skipped for experiment C48_S2SWA_gefs_6c8548e3
Case setup: Completed for experiment C48_S2SW_6c8548e3
Case setup: Completed for experiment C96_atm3DVar_extended_6c8548e3
Case setup: Skipped for experiment C96_atm3DVar_6c8548e3
Case setup: Completed for experiment C96C48_hybatmaerosnowDA_6c8548e3
Case setup: Completed for experiment C96C48_hybatmDA_6c8548e3
Case setup: Completed for experiment C96C48_ufs_hybatmDA_6c8548e3

emcbot commented 2 months ago

Experiment C48_ATM FAILED on Orion in Build# 2 with error logs:

/work2/noaa/stmp/CI/ORION/2809/RUNTESTS/COMROOT/C48_ATM_6c8548e3/logs/2021032312/gfsgenesis.log
/work2/noaa/stmp/CI/ORION/2809/RUNTESTS/COMROOT/C48_ATM_6c8548e3/logs/2021032312/gfstracker.log

Follow link here to view the contents of the above file(s): (link)

emcbot commented 2 months ago

Experiment C48_ATM FAILED on Orion in Build# 2 in /work2/noaa/stmp/CI/ORION/2809/RUNTESTS/EXPDIR/C48_ATM_6c8548e3

emcbot commented 2 months ago

Experiment C48_S2SWA_gefs FAILED on Orion in Build# 2 in /work2/noaa/stmp/CI/ORION/2809/RUNTESTS/EXPDIR/C48_S2SWA_gefs_6c8548e3

emcbot commented 2 months ago

Experiment C48_S2SW FAILED on Orion in Build# 2 with error logs:

/work2/noaa/stmp/CI/ORION/2809/RUNTESTS/COMROOT/C48_S2SW_6c8548e3/logs/2021032312/gfsgenesis.log
/work2/noaa/stmp/CI/ORION/2809/RUNTESTS/COMROOT/C48_S2SW_6c8548e3/logs/2021032312/gfstracker.log

Follow link here to view the contents of the above file(s): (link)

emcbot commented 2 months ago

Experiment C48_S2SW FAILED on Orion in Build# 2 in /work2/noaa/stmp/CI/ORION/2809/RUNTESTS/EXPDIR/C48_S2SW_6c8548e3

emcbot commented 2 months ago

Experiment C96C48_hybatmDA FAILED on Orion in Build# 2 with error logs:

/work2/noaa/stmp/CI/ORION/2809/RUNTESTS/COMROOT/C96C48_hybatmDA_6c8548e3/logs/2021122100/gfsgenesis.log
/work2/noaa/stmp/CI/ORION/2809/RUNTESTS/COMROOT/C96C48_hybatmDA_6c8548e3/logs/2021122100/gfstracker.log

Follow link here to view the contents of the above file(s): (link)

emcbot commented 2 months ago

Experiment C96C48_hybatmDA FAILED on Orion in Build# 2 in /work2/noaa/stmp/CI/ORION/2809/RUNTESTS/EXPDIR/C96C48_hybatmDA_6c8548e3

WalterKolczynski-NOAA commented 2 months ago

Ignore the Orion failure. We already know what that is.

emcbot commented 2 months ago

Experiment C96_atm3DVar FAILED on Orion in Build# 2 with error logs:

/work2/noaa/stmp/CI/ORION/2809/RUNTESTS/COMROOT/C96_atm3DVar_6c8548e3/logs/2021122100/gfsgenesis.log
/work2/noaa/stmp/CI/ORION/2809/RUNTESTS/COMROOT/C96_atm3DVar_6c8548e3/logs/2021122100/gfstracker.log

Follow link here to view the contents of the above file(s): (link)

emcbot commented 2 months ago

Experiment C96_atm3DVar FAILED on Orion in Build# 2 in /work2/noaa/stmp/CI/ORION/2809/RUNTESTS/EXPDIR/C96_atm3DVar_6c8548e3

emcbot commented 2 months ago

CI Failed on Orion in Build# 2
Built and ran in directory /work2/noaa/stmp/CI/ORION/2809



Experiment C48_ATM_6c8548e3 Terminated with 0
FAIL
FAIL tasks failed and 2 dead at Fri Sep 20 01:04:13 PM CDT 2024
Experiment C48_ATM_6c8548e3 Terminated: *FAIL*
Error logs:
/work2/noaa/stmp/CI/ORION/2809/RUNTESTS/COMROOT/C48_ATM_6c8548e3/logs/2021032312/gfsgenesis.log
/work2/noaa/stmp/CI/ORION/2809/RUNTESTS/COMROOT/C48_ATM_6c8548e3/logs/2021032312/gfstracker.log
Experiment C48_S2SWA_gefs_6c8548e3 Terminated with 0 tasks failed and 0 dead at Fri Sep 20 01:18:23 PM CDT 2024
Experiment C48_S2SWA_gefs_6c8548e3 Terminated: *UNAVAILABLE*
Experiment C48_S2SW_6c8548e3 Terminated with 0
FAIL
FAIL tasks failed and 2 dead at Fri Sep 20 01:53:16 PM CDT 2024
Experiment C48_S2SW_6c8548e3 Terminated: *FAIL*
Error logs:
/work2/noaa/stmp/CI/ORION/2809/RUNTESTS/COMROOT/C48_S2SW_6c8548e3/logs/2021032312/gfsgenesis.log
/work2/noaa/stmp/CI/ORION/2809/RUNTESTS/COMROOT/C48_S2SW_6c8548e3/logs/2021032312/gfstracker.log
Experiment C96C48_hybatmDA_6c8548e3 Terminated with 0
FAIL
FAIL tasks failed and 2 dead at Fri Sep 20 03:42:50 PM CDT 2024
Experiment C96C48_hybatmDA_6c8548e3 Terminated: *FAIL*
Error logs:
/work2/noaa/stmp/CI/ORION/2809/RUNTESTS/COMROOT/C96C48_hybatmDA_6c8548e3/logs/2021122100/gfsgenesis.log
/work2/noaa/stmp/CI/ORION/2809/RUNTESTS/COMROOT/C96C48_hybatmDA_6c8548e3/logs/2021122100/gfstracker.log
Experiment C96_atm3DVar_6c8548e3 Terminated with 0
FAIL
FAIL tasks failed and 2 dead at Fri Sep 20 10:13:54 PM CDT 2024
Experiment C96_atm3DVar_6c8548e3 Terminated: *FAIL*
Error logs:
/work2/noaa/stmp/CI/ORION/2809/RUNTESTS/COMROOT/C96_atm3DVar_6c8548e3/logs/2021122100/gfsgenesis.log
/work2/noaa/stmp/CI/ORION/2809/RUNTESTS/COMROOT/C96_atm3DVar_6c8548e3/logs/2021122100/gfstracker.log

emcbot commented 1 month ago

All CI Test Cases Passed on Wcoss2:


Experiment C48_ATM_6c8548e3 *** SUCCESS *** at 09/20/24 05:14:08 PM
Experiment C48_S2SW_6c8548e3 *** SUCCESS *** at 09/20/24 05:35:11 PM
Experiment C96C48_hybatmDA_6c8548e3 *** SUCCESS *** at 09/20/24 06:28:31 PM
Experiment C96C48_hybatmaerosnowDA_6c8548e3 *** SUCCESS *** at 09/20/24 07:00:28 PM
Experiment C96C48_ufs_hybatmDA_6c8548e3 *** SUCCESS *** at 09/20/24 07:56:23 PM
Experiment C96_atm3DVar_extended_6c8548e3 *** SUCCESS *** at 09/21/24 03:14:31 AM

emcbot commented 1 month ago

CI Passed on Hera in Build# 3
Built and ran in directory /scratch1/NCEPDEV/global/CI/2809



Experiment C48mx500_3DVarAOWCDA_6c8548e3 Completed 2 Cycles: *SUCCESS* at Fri Sep 20 20:11:45 UTC 2024
Experiment C48_ATM_6c8548e3 Completed 1 Cycles: *SUCCESS* at Fri Sep 20 22:20:29 UTC 2024
Experiment C48_S2SW_6c8548e3 Completed 1 Cycles: *SUCCESS* at Fri Sep 20 22:51:44 UTC 2024
Experiment C48_S2SWA_gefs_6c8548e3 Completed 1 Cycles: *SUCCESS* at Sat Sep 21 00:00:03 UTC 2024
Experiment C96_atm3DVar_6c8548e3 Completed 3 Cycles: *SUCCESS* at Sat Sep 21 07:37:18 UTC 2024
Experiment C96C48_hybatmDA_6c8548e3 Completed 3 Cycles: *SUCCESS* at Sat Sep 21 08:07:51 UTC 2024
Experiment C96C48_ufs_hybatmDA_6c8548e3 Completed 2 Cycles: *SUCCESS* at Sat Sep 21 11:54:22 UTC 2024
Experiment C96C48_hybatmaerosnowDA_6c8548e3 Completed 3 Cycles: *SUCCESS* at Sat Sep 21 11:54:55 UTC 2024