WalterKolczynski-NOAA commented 3 weeks ago

Description

Turns on high-frequency (hourly) output in the extended products test to exercise that aspect of the code. This test only runs on WCOSS.

Also adds the hooks to optionally turn on the metplus jobs, but that is deferred as they are not currently working correctly.

Type of change

Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO

How has this been tested?

Extended test on WCOSS

Checklist

[x] Any dependent changes have been merged and published
[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] My changes generate no new warnings
[x] New and existing tests pass with my changes
[x] I have made corresponding changes to the documentation if necessary

emcbot commented 3 weeks ago


CI Update on Wcoss2 at 06/12/24 03:16:10 AM
============================================
Cloning and Building global-workflow PR: 2679
with PID: 9287 on host: clogin05

emcbot commented 3 weeks ago

Automated global-workflow Testing Results:


Machine: Wcoss2
Start: Wed Jun 12 03:24:02 UTC 2024 on clogin05
---------------------------------------------------
Build: Completed at 06/12/24 03:58:49 AM
Case setup: Completed for experiment C48_ATM_6ad6de4a
Case setup: Skipped for experiment C48mx500_3DVarAOWCDA_6ad6de4a
Case setup: Skipped for experiment C48_S2SWA_gefs_6ad6de4a
Case setup: Completed for experiment C48_S2SW_6ad6de4a
Case setup: Completed for experiment C96_atm3DVar_extended_6ad6de4a
Case setup: Skipped for experiment C96_atm3DVar_6ad6de4a
Case setup: Skipped for experiment C96_atmaerosnowDA_6ad6de4a
Case setup: Completed for experiment C96C48_hybatmDA_6ad6de4a
Case setup: Completed for experiment C96C48_ufs_hybatmDA_6ad6de4a

emcbot commented 2 weeks ago

Experiment C48_ATM_6ad6de4a SUCCESS on Wcoss2 at 06/12/24 05:12:16 AM

emcbot commented 2 weeks ago

Experiment C48_S2SW_6ad6de4a SUCCESS on Wcoss2 at 06/12/24 05:27:15 AM

emcbot commented 2 weeks ago

Experiment C96C48_ufs_hybatmDA_6ad6de4a SUCCESS on Wcoss2 at 06/12/24 06:30:16 AM

emcbot commented 2 weeks ago

Experiment C96_atm3DVar_extended_6ad6de4a SUCCESS on Wcoss2 at 06/12/24 12:33:32 PM

emcbot commented 2 weeks ago

All CI Test Cases Passed on Wcoss2:


Experiment C48_ATM_6ad6de4a *** SUCCESS *** at 06/12/24 05:12:16 AM
Experiment C48_S2SW_6ad6de4a *** SUCCESS *** at 06/12/24 05:27:15 AM
Experiment C96C48_hybatmDA_6ad6de4a *** SUCCESS *** at 06/12/24 06:21:18 AM
Experiment C96C48_ufs_hybatmDA_6ad6de4a *** SUCCESS *** at 06/12/24 06:30:16 AM
Experiment C96_atm3DVar_extended_6ad6de4a *** SUCCESS *** at 06/12/24 12:33:32 PM

TerrenceMcGuinness-NOAA commented 2 weeks ago

Here Build failed on checkout cloning:

fatal: clone of 'https://github.com/NOAA-EMC/GSI.git' into submodule path '/scratch1/NCEPDEV/global/CI/2679/gfs/sorc/gsi_enkf.fd' failed
Failed to clone 'sorc/gsi_enkf.fd'. Retry scheduled
Cloning into '/scratch1/NCEPDEV/global/CI/2679/gfs/sorc/gsi_enkf.fd'...
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
error: 4431 bytes of body are still expected
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output
fatal: clone of 'https://github.com/NOAA-EMC/GSI.git' into submodule path '/scratch1/NCEPDEV/global/CI/2[67](https://jenkins.epic.oarcloud.noaa.gov/job/global-workflow/job/EMC-Global-Pipeline/view/change-requests/job/PR-2679/2/pipeline-console/?start-byte=0&selected-node=74#log-67)9/gfs/sorc/gsi_enkf.fd' failed
Failed to clone 'sorc/gsi_enkf.fd' a second time, aborting

WalterKolczynski-NOAA commented 2 weeks ago

Will wait a bit and try again

emcbot commented 2 weeks ago

CI Passed Hera at
Built and ran in directory /scratch1/NCEPDEV/global/CI/2679

emcbot commented 2 weeks ago


CI Update on Wcoss2 at 06/14/24 03:15:10 AM
============================================
Cloning and Building global-workflow PR: 2679
with PID: 230344 on host: dlogin08

emcbot commented 2 weeks ago

Failed on cloning and building global-workflowi PR: 2679 CI on Wcoss2 failed to build on Fri Jun 14 03:15:24 UTC 2024 for repo git@github.com:NOAA-EMC/global-workflow.git

emcbot commented 2 weeks ago

Experiment C96C48_hybatmDA FAILED on Hercules with error logs:

/work2/noaa/stmp/CI/HERCULES/2679/RUNTESTS/COMROOT/C96C48_hybatmDA_ec57d74a/logs/2021122100/gfsatmos_prod_f090-f096.log

Follow link here to view the contents of the above file(s): (link)

emcbot commented 2 weeks ago

Experiment C96C48_hybatmDA FAILED on Hercules in /work2/noaa/stmp/CI/HERCULES/2679/RUNTESTS/C96C48_hybatmDA_ec57d74a

emcbot commented 2 weeks ago

Experiment C96_atm3DVar FAILED on Hercules with error logs:

/work2/noaa/stmp/CI/HERCULES/2679/RUNTESTS/COMROOT/C96_atm3DVar_ec57d74a/logs/2021122100/gfsatmos_prod_f099-f105.log

Follow link here to view the contents of the above file(s): (link)

emcbot commented 2 weeks ago

Experiment C96_atm3DVar FAILED on Hercules in /work2/noaa/stmp/CI/HERCULES/2679/RUNTESTS/C96_atm3DVar_ec57d74a

DavidHuber-NOAA commented 2 weeks ago

The gfsatmos_prod_f099-f105 job failed on Hercules due to the job taking too long. I suspect a bad node and will reboot the testing.

DavidHuber-NOAA commented 2 weeks ago

Checking manually, all tests passed on Hera, removing the tag.

TerrenceMcGuinness-NOAA commented 2 weeks ago

@DavidHuber-NOAA No C96_atmaerosnoDA on Hera is still running. Put label back to running .emcbot will mark if SUCCESS when archive is done in just a sec.

emcbot commented 2 weeks ago

CI Passed Hera at
Built and ran in directory /scratch1/NCEPDEV/global/CI/2679

emcbot commented 2 weeks ago

CI Passed Hercules at
Built and ran in directory /work2/noaa/stmp/CI/HERCULES/2679

DavidHuber-NOAA commented 2 weeks ago

Great, thanks @TerrenceMcGuinness-NOAA!

emcbot commented 2 weeks ago


CI Update on Wcoss2 at 06/17/24 05:21:10 PM
============================================
Cloning and Building global-workflow PR: 2679
with PID: 233992 on host: dlogin08

emcbot commented 2 weeks ago

Automated global-workflow Testing Results:


Machine: Wcoss2
Start: Mon Jun 17 17:25:40 UTC 2024 on dlogin08
---------------------------------------------------
Build: Completed at 06/17/24 06:07:50 PM
Case setup: Completed for experiment C48_ATM_ec57d74a
Case setup: Skipped for experiment C48mx500_3DVarAOWCDA_ec57d74a
Case setup: Skipped for experiment C48_S2SWA_gefs_ec57d74a
Case setup: Completed for experiment C48_S2SW_ec57d74a
Case setup: Completed for experiment C96_atm3DVar_extended_ec57d74a
Case setup: Skipped for experiment C96_atm3DVar_ec57d74a
Case setup: Skipped for experiment C96_atmaerosnowDA_ec57d74a
Case setup: Completed for experiment C96C48_hybatmDA_ec57d74a
Case setup: Completed for experiment C96C48_ufs_hybatmDA_ec57d74a

emcbot commented 2 weeks ago

Experiment C48_ATM_ec57d74a SUCCESS on Wcoss2 at 06/17/24 07:16:18 PM

emcbot commented 2 weeks ago

Experiment C48_S2SW_ec57d74a SUCCESS on Wcoss2 at 06/17/24 07:52:22 PM

emcbot commented 2 weeks ago

Experiment C96C48_hybatmDA_ec57d74a SUCCESS on Wcoss2 at 06/17/24 08:32:23 PM

emcbot commented 2 weeks ago

Experiment C96C48_ufs_hybatmDA_ec57d74a SUCCESS on Wcoss2 at 06/17/24 08:36:25 PM

emcbot commented 2 weeks ago

Experiment C96_atm3DVar_extended_ec57d74a SUCCESS on Wcoss2 at 06/18/24 02:44:35 AM

emcbot commented 2 weeks ago

All CI Test Cases Passed on Wcoss2:


Experiment C48_ATM_ec57d74a *** SUCCESS *** at 06/17/24 07:16:18 PM
Experiment C48_S2SW_ec57d74a *** SUCCESS *** at 06/17/24 07:52:22 PM
Experiment C96C48_hybatmDA_ec57d74a *** SUCCESS *** at 06/17/24 08:32:23 PM
Experiment C96C48_ufs_hybatmDA_ec57d74a *** SUCCESS *** at 06/17/24 08:36:25 PM
Experiment C96_atm3DVar_extended_ec57d74a *** SUCCESS *** at 06/18/24 02:44:35 AM

NOAA-EMC / global-workflow

Turn on high-frequency output in extended test #2679

Description

Type of change

Change characteristics

How has this been tested?

Checklist