NOAA-EMC / global-workflow

Global Superstructure/Workflow supporting the Global Forecast System (GFS)
https://global-workflow.readthedocs.io/en/latest
GNU Lesser General Public License v3.0
70 stars 162 forks source link

Update cleanup job to use COMIN/COMOUT #2649

Closed HenryRWinterbottom closed 3 weeks ago

HenryRWinterbottom commented 1 month ago

Description

NCO has requested that each COM variable specify whether it is an input or an output. This completes that process for the global-workflow clean-up task.

Refs #2451

Type of change

Change characteristics

How has this been tested?

Change characteristics

How has this been tested?

       CYCLE                    TASK                       JOBID               STATE         EXIT STATUS     TRIES      DURATION
================================================================================================================================
202103231200             gfsstage_ic                    61083306           SUCCEEDED                   0         1          22.0
202103231200             gfswaveinit                    61083307           SUCCEEDED                   0         1          31.0
202103231200                 gfsfcst                    61083520           SUCCEEDED                   0         1        2299.0
202103231200    gfsatmos_prod_f000-f006                    61083820           SUCCEEDED                   0         1         426.0
202103231200    gfsatmos_prod_f009-f015                    61083821           SUCCEEDED                   0         1         434.0
202103231200    gfsatmos_prod_f018-f024                    61083878           SUCCEEDED                   0         1         381.0
202103231200    gfsatmos_prod_f027-f033                    61083879           SUCCEEDED                   0         1         384.0
202103231200    gfsatmos_prod_f036-f042                    61084008           SUCCEEDED                   0         1         390.0
202103231200    gfsatmos_prod_f045-f051                    61084009           SUCCEEDED                   0         1         395.0
202103231200    gfsatmos_prod_f054-f060                    61084148           SUCCEEDED                   0         1         550.0
202103231200    gfsatmos_prod_f063-f069                    61084332           SUCCEEDED                   0         1         493.0
202103231200    gfsatmos_prod_f072-f078                    61084333           SUCCEEDED                   0         1         477.0
202103231200    gfsatmos_prod_f081-f087                    61084389           SUCCEEDED                   0         1         397.0
202103231200    gfsatmos_prod_f090-f096                    61084390           SUCCEEDED                   0         1         394.0
202103231200    gfsatmos_prod_f099-f105                    61084571           SUCCEEDED                   0         1         392.0
202103231200    gfsatmos_prod_f108-f114                    61084559           SUCCEEDED                   0         1         388.0
202103231200    gfsatmos_prod_f117-f120                    61084718           SUCCEEDED                   0         1         258.0
202103231200              gfstracker                    61084973           SUCCEEDED                   0         1          81.0
202103231200              gfsgenesis                    61084974           SUCCEEDED                   0         1         541.0
202103231200    gfsocean_prod_f006-f018                    61083924           SUCCEEDED                   0         1          79.0
202103231200    gfsocean_prod_f024-f036                    61084049           SUCCEEDED                   0         1          83.0
202103231200    gfsocean_prod_f042-f054                    61084181           SUCCEEDED                   0         1          93.0
202103231200    gfsocean_prod_f060-f072                    61084336           SUCCEEDED                   0         1         155.0
202103231200    gfsocean_prod_f078-f090                    61084471           SUCCEEDED                   0         1          84.0
202103231200    gfsocean_prod_f096-f108                    61084597           SUCCEEDED                   0         1          89.0
202103231200    gfsocean_prod_f114-f120                    61084598           SUCCEEDED                   0         1          55.0
202103231200    gfsice_prod_f006-f018                    61083925           SUCCEEDED                   0         1          59.0
202103231200    gfsice_prod_f024-f036                    61084050           SUCCEEDED                   0         1          63.0
202103231200    gfsice_prod_f042-f054                    61084182           SUCCEEDED                   0         1          62.0
202103231200    gfsice_prod_f060-f072                    61084337           SUCCEEDED                   0         1          86.0
202103231200    gfsice_prod_f078-f090                    61084472           SUCCEEDED                   0         1          69.0
202103231200    gfsice_prod_f096-f108                    61084599           SUCCEEDED                   0         1          63.0
202103231200    gfsice_prod_f114-f120                    61084600           SUCCEEDED                   0         1          44.0
202103231200          gfswavepostsbs                    61083681           SUCCEEDED                   0         1        2427.0
202103231200          gfswavepostpnt                    61084601           SUCCEEDED                   0         1       11260.0
202103231200                 gfsarch                    61092509           SUCCEEDED                   0         1          48.0
202103231200              gfscleanup                    61188921           SUCCEEDED                   0         1          26.0
================================================================================================================================
202103241200             gfsstage_ic                          -1           SUCCEEDED                   0         1           0.0
202103241200             gfswaveinit                          -1           SUCCEEDED                   0         1           0.0
202103241200                 gfsfcst                          -1           SUCCEEDED                   0         1           0.0
202103241200    gfsatmos_prod_f000-f006                          -1           SUCCEEDED                   0         1           0.0
202103241200    gfsatmos_prod_f009-f015                          -1           SUCCEEDED                   0         1           0.0
202103241200    gfsatmos_prod_f018-f024                          -1           SUCCEEDED                   0         1           0.0
202103241200    gfsatmos_prod_f027-f033                          -1           SUCCEEDED                   0         1           0.0
202103241200    gfsatmos_prod_f036-f042                          -1           SUCCEEDED                   0         1           0.0
202103241200    gfsatmos_prod_f045-f051                          -1           SUCCEEDED                   0         1           0.0
202103241200    gfsatmos_prod_f054-f060                          -1           SUCCEEDED                   0         1           0.0
202103241200    gfsatmos_prod_f063-f069                          -1           SUCCEEDED                   0         1           0.0
202103241200    gfsatmos_prod_f072-f078                          -1           SUCCEEDED                   0         1           0.0
202103241200    gfsatmos_prod_f081-f087                          -1           SUCCEEDED                   0         1           0.0
202103241200    gfsatmos_prod_f090-f096                          -1           SUCCEEDED                   0         1           0.0
202103241200    gfsatmos_prod_f099-f105                          -1           SUCCEEDED                   0         1           0.0
202103241200    gfsatmos_prod_f108-f114                          -1           SUCCEEDED                   0         1           0.0
202103241200    gfsatmos_prod_f117-f120                          -1           SUCCEEDED                   0         1           0.0
202103241200              gfstracker                          -1           SUCCEEDED                   0         1           0.0
202103241200              gfsgenesis                          -1           SUCCEEDED                   0         1           0.0
202103241200    gfsocean_prod_f006-f018                          -1           SUCCEEDED                   0         1           0.0
202103241200    gfsocean_prod_f024-f036                          -1           SUCCEEDED                   0         1           0.0
202103241200    gfsocean_prod_f042-f054                          -1           SUCCEEDED                   0         1           0.0
202103241200    gfsocean_prod_f060-f072                          -1           SUCCEEDED                   0         1           0.0
202103241200    gfsocean_prod_f078-f090                          -1           SUCCEEDED                   0         1           0.0
202103241200    gfsocean_prod_f096-f108                          -1           SUCCEEDED                   0         1           0.0
202103241200    gfsocean_prod_f114-f120                          -1           SUCCEEDED                   0         1           0.0
202103241200    gfsice_prod_f006-f018                          -1           SUCCEEDED                   0         1           0.0
202103241200    gfsice_prod_f024-f036                          -1           SUCCEEDED                   0         1           0.0
202103241200    gfsice_prod_f042-f054                          -1           SUCCEEDED                   0         1           0.0
202103241200    gfsice_prod_f060-f072                          -1           SUCCEEDED                   0         1           0.0
202103241200    gfsice_prod_f078-f090                          -1           SUCCEEDED                   0         1           0.0
202103241200    gfsice_prod_f096-f108                          -1           SUCCEEDED                   0         1           0.0
202103241200    gfsice_prod_f114-f120                          -1           SUCCEEDED                   0         1           0.0
202103241200          gfswavepostsbs                          -1           SUCCEEDED                   0         1           0.0
202103241200          gfswavepostpnt                          -1           SUCCEEDED                   0         1           0.0
202103241200                 gfsarch                          -1           SUCCEEDED                   0         1           0.0
202103241200              gfscleanup                    61188920           SUCCEEDED                   0         1          27.0

Checklist

emcbot commented 4 weeks ago

CI Update on Wcoss2 at 06/04/24 02:32:12 PM
============================================
Cloning and Building global-workflow PR: 2649
with PID: 92236 on host: clogin01
emcbot commented 4 weeks ago

Automated global-workflow Testing Results:


Machine: Wcoss2
Start: Tue Jun  4 14:42:07 UTC 2024 on clogin01
---------------------------------------------------
Build: Completed at 06/04/24 02:54:51 PM
Case setup: Completed for experiment C48_ATM_061e13eb
Case setup: Skipped for experiment C48mx500_3DVarAOWCDA_061e13eb
Case setup: Skipped for experiment C48_S2SWA_gefs_061e13eb
Case setup: Completed for experiment C48_S2SW_061e13eb
Case setup: Completed for experiment C96_atm3DVar_extended_061e13eb
Case setup: Skipped for experiment C96_atm3DVar_061e13eb
Case setup: Skipped for experiment C96_atmaerosnowDA_061e13eb
Case setup: Completed for experiment C96C48_hybatmDA_061e13eb
Case setup: Skipped for experiment C96C48_ufs_hybatmDA_061e13eb
emcbot commented 4 weeks ago

Experiment C48_ATM_061e13eb SUCCESS on Wcoss2 at 06/04/24 04:00:17 PM

emcbot commented 4 weeks ago

Experiment C48_S2SW_061e13eb SUCCESS on Wcoss2 at 06/04/24 04:15:10 PM

emcbot commented 4 weeks ago

Experiment C96C48_hybatmDA_061e13eb SUCCESS on Wcoss2 at 06/04/24 05:00:24 PM

emcbot commented 4 weeks ago

Experiment C96_atm3DVar_extended_061e13eb **** on Wcoss2 at 06/04/24 09:45:09 PM

Error logs:

Follow link here to view the contents of the above file(s): [(link)]()

emcbot commented 4 weeks ago

CI Passed Orion at
Built and ran in directory /work2/noaa/stmp/CI/ORION/2649

emcbot commented 3 weeks ago

CI Update on Wcoss2 at 06/05/24 07:48:55 PM
============================================
Cloning and Building global-workflow PR: 2649
with PID: 40491 on host: clogin05
emcbot commented 3 weeks ago

Automated global-workflow Testing Results:


Machine: Wcoss2
Start: Wed Jun  5 19:58:20 UTC 2024 on clogin05
---------------------------------------------------
Build: Completed at 06/05/24 08:56:50 PM
Case setup: Completed for experiment C48_ATM_9b290b28
Case setup: Skipped for experiment C48mx500_3DVarAOWCDA_9b290b28
Case setup: Skipped for experiment C48_S2SWA_gefs_9b290b28
Case setup: Completed for experiment C48_S2SW_9b290b28
Case setup: Completed for experiment C96_atm3DVar_extended_9b290b28
Case setup: Skipped for experiment C96_atm3DVar_9b290b28
Case setup: Skipped for experiment C96_atmaerosnowDA_9b290b28
Case setup: Completed for experiment C96C48_hybatmDA_9b290b28
Case setup: Completed for experiment C96C48_ufs_hybatmDA_9b290b28
emcbot commented 3 weeks ago

Experiment C48_S2SW_9b290b28 FAIL on Wcoss2 at 06/05/24 10:12:27 PM

Error logs:

/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2649/RUNTESTS/COMROOT/C48_S2SW_9b290b28/logs/2021032312/gfswavepostpnt.log

Follow link here to view the contents of the above file(s): (link)

emcbot commented 3 weeks ago

CI Passed Hercules at
Built and ran in directory /work2/noaa/stmp/CI/HERCULES/2649

emcbot commented 3 weeks ago

CI Passed Hera at
Built and ran in directory /scratch1/NCEPDEV/global/CI/2649

WalterKolczynski-NOAA commented 3 weeks ago

stmp filled on WCOSS again.

emcbot commented 3 weeks ago

CI Update on Wcoss2 at 06/06/24 05:48:58 AM
============================================
Cloning and Building global-workflow PR: 2649
with PID: 5340 on host: clogin05
emcbot commented 3 weeks ago

Automated global-workflow Testing Results:


Machine: Wcoss2
Start: Thu Jun  6 05:56:20 UTC 2024 on clogin05
---------------------------------------------------
Build: Completed at 06/06/24 06:32:06 AM
Case setup: Completed for experiment C48_ATM_9b290b28
Case setup: Skipped for experiment C48mx500_3DVarAOWCDA_9b290b28
Case setup: Skipped for experiment C48_S2SWA_gefs_9b290b28
Case setup: Completed for experiment C48_S2SW_9b290b28
Case setup: Completed for experiment C96_atm3DVar_extended_9b290b28
Case setup: Skipped for experiment C96_atm3DVar_9b290b28
Case setup: Skipped for experiment C96_atmaerosnowDA_9b290b28
Case setup: Completed for experiment C96C48_hybatmDA_9b290b28
Case setup: Completed for experiment C96C48_ufs_hybatmDA_9b290b28
emcbot commented 3 weeks ago

Experiment C96_atm3DVar_extended_9b290b28 FAIL on Wcoss2 at 06/06/24 07:36:27 AM

Error logs:

/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2649/RUNTESTS/COMROOT/C96_atm3DVar_extended_9b290b28/logs/2021122100/gfsatmos_prod_f009-f015.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2649/RUNTESTS/COMROOT/C96_atm3DVar_extended_9b290b28/logs/2021122100/gfsgoesupp_f009-f015.log

Follow link here to view the contents of the above file(s): (link)

WalterKolczynski-NOAA commented 3 weeks ago

Failure is unrelated to the PR. We need Terry to clean out the restart directory when he gets back Monday.

emcbot commented 3 weeks ago

CI Update on Wcoss2 at 06/11/24 03:17:07 PM
============================================
Cloning and Building global-workflow PR: 2649
with PID: 237538 on host: clogin05
emcbot commented 3 weeks ago

Failed on cloning and building global-workflowi PR: 2649 CI on Wcoss2 failed to build on Tue Jun 11 15:17:53 UTC 2024 for repo git@github.com:NOAA-EMC/global-workflow.git

emcbot commented 3 weeks ago

CI Update on Wcoss2 at 06/11/24 04:48:11 PM
============================================
Cloning and Building global-workflow PR: 2649
with PID: 141663 on host: clogin05
emcbot commented 3 weeks ago

Automated global-workflow Testing Results:


Machine: Wcoss2
Start: Tue Jun 11 16:56:10 UTC 2024 on clogin05
---------------------------------------------------
Build: Completed at 06/11/24 05:48:50 PM
Case setup: Completed for experiment C48_ATM_81103f6f
Case setup: Skipped for experiment C48mx500_3DVarAOWCDA_81103f6f
Case setup: Skipped for experiment C48_S2SWA_gefs_81103f6f
Case setup: Completed for experiment C48_S2SW_81103f6f
Case setup: Completed for experiment C96_atm3DVar_extended_81103f6f
Case setup: Skipped for experiment C96_atm3DVar_81103f6f
Case setup: Skipped for experiment C96_atmaerosnowDA_81103f6f
Case setup: Completed for experiment C96C48_hybatmDA_81103f6f
Case setup: Completed for experiment C96C48_ufs_hybatmDA_81103f6f
emcbot commented 3 weeks ago

Experiment C48_S2SW_81103f6f SUCCESS on Wcoss2 at 06/11/24 07:21:16 PM

emcbot commented 3 weeks ago

Experiment C96C48_ufs_hybatmDA_81103f6f SUCCESS on Wcoss2 at 06/11/24 08:03:18 PM

emcbot commented 3 weeks ago

Experiment C96_atm3DVar_extended_81103f6f SUCCESS on Wcoss2 at 06/12/24 01:51:26 AM

emcbot commented 3 weeks ago

All CI Test Cases Passed on Wcoss2:


Experiment C48_ATM_81103f6f *** SUCCESS *** at 06/11/24 07:00:15 PM
Experiment C48_S2SW_81103f6f *** SUCCESS *** at 06/11/24 07:21:16 PM
Experiment C96C48_hybatmDA_81103f6f *** SUCCESS *** at 06/11/24 07:57:20 PM
Experiment C96C48_ufs_hybatmDA_81103f6f *** SUCCESS *** at 06/11/24 08:03:18 PM
Experiment C96_atm3DVar_extended_81103f6f *** SUCCESS *** at 06/12/24 01:51:26 AM