JessicaMeixner-NOAA commented 3 weeks ago

to work while spack inconsistent bt workflow and ww3/ufs

Description

Updates ufs-weather-model, this updates RDHPCS to the newer spack-stack allowing some temporary fixes to be reverted.

removes upp submodule
uses upp from the ufs-weather-model
restores the build and link that were hacked during the Hera Rocky 8 transition to allow for UPP submodule
Removes forecast directories in clean-up

Fixes https://github.com/NOAA-EMC/global-workflow/issues/2617 Fixes https://github.com/NOAA-EMC/global-workflow/issues/2437

co-author: @aerorahul

Type of change

Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO

How has this been tested?

In progress...

Checklist

[ ] Any dependent changes have been merged and published
[ ] My code follows the style guidelines of this project
[ ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] My changes generate no new warnings
[ ] New and existing tests pass with my changes
[ ] I have made corresponding changes to the documentation if necessary

aerorahul commented 3 weeks ago

Thanks for this PR @JessicaMeixner-NOAA I have opened a PR into this PR branch https://github.com/JessicaMeixner-NOAA/global-workflow/pull/13 to remove the UPP submodule and use the UPP directly from the UFS-weather-model. This will resolve #2437

JessicaMeixner-NOAA commented 3 weeks ago

Thanks @aerorahul ! I just merged this in. I'll take the PR off of draft but I don't know if you want to wait for running full set of tests until after I've done an offline test or not?

JessicaMeixner-NOAA commented 3 weeks ago

S2SW test on hera is currently going through the wave post point.

emcbot commented 3 weeks ago


CI Update on Wcoss2 at 06/07/24 05:08:10 PM
============================================
Cloning and Building global-workflow PR: 2663
with PID: 201574 on host: clogin05

emcbot commented 3 weeks ago

Automated global-workflow Testing Results:


Machine: Wcoss2
Start: Fri Jun  7 17:16:38 UTC 2024 on clogin05
---------------------------------------------------
Build: Completed at 06/07/24 05:53:08 PM
Case setup: Completed for experiment C48_ATM_ccd87d4f
Case setup: Skipped for experiment C48mx500_3DVarAOWCDA_ccd87d4f
Case setup: Skipped for experiment C48_S2SWA_gefs_ccd87d4f
Case setup: Completed for experiment C48_S2SW_ccd87d4f
Case setup: Completed for experiment C96_atm3DVar_extended_ccd87d4f
Case setup: Skipped for experiment C96_atm3DVar_ccd87d4f
Case setup: Skipped for experiment C96_atmaerosnowDA_ccd87d4f
Case setup: Completed for experiment C96C48_hybatmDA_ccd87d4f
Case setup: Completed for experiment C96C48_ufs_hybatmDA_ccd87d4f

emcbot commented 3 weeks ago

Experiment C48_ATM_ccd87d4f SUCCESS on Wcoss2 at 06/07/24 07:18:15 PM

emcbot commented 3 weeks ago

Experiment C48_S2SW_ccd87d4f SUCCESS on Wcoss2 at 06/07/24 07:42:13 PM

emcbot commented 3 weeks ago

Experiment C96C48_hybatmDA FAILED on Orion with error logs:

/work2/noaa/stmp/CI/ORION/2663/RUNTESTS/COMROOT/C96C48_hybatmDA_ccd87d4f/logs/2021122100/gfsatmos_prod_f063-f069.log

Follow link here to view the contents of the above file(s): (link)

emcbot commented 3 weeks ago

Experiment C96C48_hybatmDA FAILED on Orion in /work2/noaa/stmp/CI/ORION/2663/RUNTESTS/C96C48_hybatmDA_ccd87d4f

WalterKolczynski-NOAA commented 3 weeks ago

Hit the wallclock on Orion in one product job. Maybe a bad node?

emcbot commented 3 weeks ago

Experiment C96C48_hybatmDA_ccd87d4f SUCCESS on Wcoss2 at 06/07/24 09:27:19 PM

emcbot commented 3 weeks ago

Experiment C96C48_ufs_hybatmDA_ccd87d4f SUCCESS on Wcoss2 at 06/07/24 09:27:24 PM

emcbot commented 3 weeks ago

Experiment C48_S2SW FAILED on Hera with error logs:

/scratch1/NCEPDEV/global/CI/2663/RUNTESTS/COMROOT/C48_S2SW_ccd87d4f/logs/2021032312/gfsarch.log

Follow link here to view the contents of the above file(s): (link)

emcbot commented 3 weeks ago

Experiment C48_S2SW FAILED on Hera in /scratch1/NCEPDEV/global/CI/2663/RUNTESTS/C48_S2SW_ccd87d4f

JessicaMeixner-NOAA commented 3 weeks ago

The hera archive failure is probably related to the update we made for the ocean daily files. I'll look into this on Monday.

emcbot commented 3 weeks ago

Experiment C96_atm3DVar_extended_ccd87d4f SUCCESS on Wcoss2 at 06/08/24 02:27:32 AM

emcbot commented 3 weeks ago

All CI Test Cases Passed on Wcoss2:


Experiment C48_ATM_ccd87d4f *** SUCCESS *** at 06/07/24 07:18:15 PM
Experiment C48_S2SW_ccd87d4f *** SUCCESS *** at 06/07/24 07:42:13 PM
Experiment C96C48_hybatmDA_ccd87d4f *** SUCCESS *** at 06/07/24 09:27:19 PM
Experiment C96C48_ufs_hybatmDA_ccd87d4f *** SUCCESS *** at 06/07/24 09:27:24 PM
Experiment C96_atm3DVar_extended_ccd87d4f *** SUCCESS *** at 06/08/24 02:27:32 AM

JessicaMeixner-NOAA commented 3 weeks ago

So the behavior of the dates of the ocean daily file is different it appears with and without IAU. I'm not sure what the best way to fix this is. Is it adding a "if IAU" part to the ocean daily linking? If we move to a true "daily" instead of the 24 hour average, its my understanding we'd have to start the averaging at 00z (or if 06, 12, 18z that would mean the next 00z), but that might at least be consistent?

If we want the updates of the model ASAP, we can take out the change we made for ocean daily files here or make the archive not be required to have the daily files so it wont fail (which also might not be ideal, since that identified an issue we need to be aware of).

WalterKolczynski-NOAA commented 3 weeks ago

So the behavior of the dates of the ocean daily file is different it appears with and without IAU. I'm not sure what the best way to fix this is. Is it adding a "if IAU" part to the ocean daily linking? If we move to a true "daily" instead of the 24 hour average, its my understanding we'd have to start the averaging at 00z (or if 06, 12, 18z that would mean the next 00z), but that might at least be consistent?

If we want the updates of the model ASAP, we can take out the change we made for ocean daily files here or make the archive not be required to have the daily files so it wont fail (which also might not be ideal, since that identified an issue we need to be aware of).

I would favor taking it out since it is separate from the model update. It can be handled in a separate PR.

JessicaMeixner-NOAA commented 3 weeks ago

Sounds good @WalterKolczynski-NOAA I'll update this PR now.

JessicaMeixner-NOAA commented 3 weeks ago

Okay, i have updated the PR and PR description as we will not be addressing the last ocn daily issue.

emcbot commented 3 weeks ago

Experiment C48mx500_3DVarAOWCDA FAILED on Hera with error logs:

/scratch1/NCEPDEV/global/CI/2663/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_8e3156c4/logs/2021032418/gdasocnanalbmat.log

Follow link here to view the contents of the above file(s): (link)

emcbot commented 3 weeks ago

Experiment C48mx500_3DVarAOWCDA FAILED on Hera in /scratch1/NCEPDEV/global/CI/2663/RUNTESTS/C48mx500_3DVarAOWCDA_8e3156c4

emcbot commented 3 weeks ago

Experiment C96_atmaerosnowDA FAILED on Hera with error logs:

/scratch1/NCEPDEV/global/CI/2663/RUNTESTS/COMROOT/C96_atmaerosnowDA_8e3156c4/logs/2021122018/gdasprepsnowobs.log

Follow link here to view the contents of the above file(s): (link)

emcbot commented 3 weeks ago

Experiment C96_atmaerosnowDA FAILED on Hera in /scratch1/NCEPDEV/global/CI/2663/RUNTESTS/C96_atmaerosnowDA_8e3156c4

WalterKolczynski-NOAA commented 3 weeks ago

Something happened with the GDAS build on Hera. Not sure what, but executables are missing.

WalterKolczynski-NOAA commented 3 weeks ago

Pretty sure unrelated to this PR. Will try again in a few minutes.

emcbot commented 3 weeks ago

CI Passed Hercules at
Built and ran in directory /work2/noaa/stmp/CI/HERCULES/2663

emcbot commented 2 weeks ago

CI Passed Hera at
Built and ran in directory /scratch1/NCEPDEV/global/CI/2663

NOAA-EMC / global-workflow

Update ufs-weather-model #2663

Description

Type of change

Change characteristics

How has this been tested?

Checklist