Update STMP and PTMP settings in host file for Orion and Hercules

[TerrenceMcGuinness-NOAA]

Description

• Updating STMP and PTMP settings in host file for Orion and Hercules because they are cross mounted.
• Also took the opportunity to finally update SLURM_ACCOUNT to HPC_ACCOUT in CI over rides.
• Added a refactor of the rocotostat.py tool that is more pythonic and as a execute retry feature because the rocotostat utility on Orion has been failing sometimes.

Type of change

• Bug fix (fixes something broken)

Change characteristics

• Is this a breaking change (a change in existing functionality)? NO
• Does this change require a documentation update? NO

[emcbot]

CI Update on Wcoss2 at 05/21/24 10:16:17 PM
============================================
Cloning and Building global-workflow PR: 2614
with PID: 19411 on host: clogin05

[emcbot]

Automated global-workflow Testing Results:

Machine: Wcoss2
Start: Tue May 21 22:27:16 UTC 2024 on clogin05
---------------------------------------------------
Build: Completed at 05/21/24 10:38:51 PM
Case setup: Completed for experiment C48_ATM_b989785e
Case setup: Skipped for experiment C48mx500_3DVarAOWCDA_b989785e
Case setup: Skipped for experiment C48_S2SWA_gefs_b989785e
Case setup: Completed for experiment C48_S2SW_b989785e
Case setup: Completed for experiment C96_atm3DVar_extended_b989785e
Case setup: Skipped for experiment C96_atm3DVar_b989785e
Case setup: Skipped for experiment C96_atmaerosnowDA_b989785e
Case setup: Completed for experiment C96C48_hybatmDA_b989785e
Case setup: Skipped for experiment C96C48_ufs_hybatmDA_b989785e

[emcbot]

Experiment C48_ATM FAILED on Orion in /work2/noaa/stmp/CI/ORION/2614/RUNTESTS/C48_ATM_cc02f5f0

[emcbot]

Experiment C48_S2SWA_gefs FAILED on Orion in /work2/noaa/stmp/CI/ORION/2614/RUNTESTS/C48_S2SWA_gefs_cc02f5f0

[emcbot]

Experiment C96_atm3DVar FAILED on Orion in /work2/noaa/stmp/CI/ORION/2614/RUNTESTS/C96_atm3DVar_cc02f5f0

[emcbot]

Experiment C96C48_hybatmDA FAILED on Orion in /work2/noaa/stmp/CI/ORION/2614/RUNTESTS/C96C48_hybatmDA_cc02f5f0

[emcbot]

Experiment C48_S2SW FAILED on Orion in /work2/noaa/stmp/CI/ORION/2614/RUNTESTS/C48_S2SW_cc02f5f0

[DavidHuber-NOAA]

Looks like Orion's rocoto server glitched like it did for Hera a few days ago.

[emcbot]

CI Passed Hercules at
Built and ran in directory /work2/noaa/stmp/CI/HERCULES/2614

[emcbot]

Experiment C96_atm3DVar FAILED on Orion in /work2/noaa/stmp/CI/ORION/2614/RUNTESTS/C96_atm3DVar_cc02f5f0

[emcbot]

Experiment C48_S2SWA_gefs FAILED on Orion in /work2/noaa/stmp/CI/ORION/2614/RUNTESTS/C48_S2SWA_gefs_cc02f5f0

[emcbot]

Experiment C48_S2SW FAILED on Orion in /work2/noaa/stmp/CI/ORION/2614/RUNTESTS/C48_S2SW_cc02f5f0

[emcbot]

Experiment C96C48_hybatmDA FAILED on Orion in /work2/noaa/stmp/CI/ORION/2614/RUNTESTS/C96C48_hybatmDA_cc02f5f0

[emcbot]

Experiment C48_ATM FAILED on Orion in /work2/noaa/stmp/CI/ORION/2614/RUNTESTS/C48_ATM_f41b4d6b

[emcbot]

Experiment C48_S2SW FAILED on Orion in /work2/noaa/stmp/CI/ORION/2614/RUNTESTS/C48_S2SW_f41b4d6b

[emcbot]

Experiment C96C48_hybatmDA FAILED on Orion in /work2/noaa/stmp/CI/ORION/2614/RUNTESTS/C96C48_hybatmDA_f41b4d6b

[emcbot]

Experiment C96_atm3DVar FAILED on Orion in /work2/noaa/stmp/CI/ORION/2614/RUNTESTS/C96_atm3DVar_f41b4d6b

[emcbot]

Experiment C48_S2SWA_gefs FAILED on Orion in /work2/noaa/stmp/CI/ORION/2614/RUNTESTS/C48_S2SWA_gefs_f41b4d6b

[emcbot]

CI Passed Hera at
Built and ran in directory /scratch1/NCEPDEV/global/CI/2614

[emcbot]

Experiment C48_S2SWA_gefs FAILED on Hercules in /work2/noaa/stmp/CI/HERCULES/2614/RUNTESTS/C48_S2SWA_gefs_f41b4d6b

[emcbot]

Experiment C48_ATM FAILED on Hercules in /work2/noaa/stmp/CI/HERCULES/2614/RUNTESTS/C48_ATM_f41b4d6b

[emcbot]

Experiment C48_S2SW FAILED on Hercules in /work2/noaa/stmp/CI/HERCULES/2614/RUNTESTS/C48_S2SW_f41b4d6b

[emcbot]

Experiment C96C48_hybatmDA FAILED on Hercules in /work2/noaa/stmp/CI/HERCULES/2614/RUNTESTS/C96C48_hybatmDA_f41b4d6b

[emcbot]

Experiment C96_atm3DVar FAILED on Hercules in /work2/noaa/stmp/CI/HERCULES/2614/RUNTESTS/C96_atm3DVar_f41b4d6b

[TerrenceMcGuinness-NOAA]

Everything looked ok in place. Made some changes in wait times and added a bug fix on STALL check and retrying.

[emcbot]

CI Update on Wcoss2 at 05/23/24 02:48:07 PM
=================================================
PR:2614 Reset to Wcoss2-Ready by user and is now restarting CI tests
Canceled all jobs for experiment C48_S2SW_b989785e in PR:2614 on Wcoss2
Canceled all jobs for experiment C96C48_hybatmDA_b989785e in PR:2614 on Wcoss2
Canceled all jobs for experiment C48_ATM_b989785e in PR:2614 on Wcoss2
Canceled all jobs for experiment C96_atm3DVar_extended_b989785e in PR:2614 on Wcoss2

[emcbot]

CI Update on Wcoss2 at 05/23/24 02:48:40 PM
============================================
Cloning and Building global-workflow PR: 2614
with PID: 235439 on host: clogin05

[emcbot]

Automated global-workflow Testing Results:

Machine: Wcoss2
Start: Thu May 23 14:57:58 UTC 2024 on clogin05
---------------------------------------------------
Build: Completed at 05/23/24 03:09:20 PM
Case setup: Completed for experiment C48_ATM_c711040b
Case setup: Skipped for experiment C48mx500_3DVarAOWCDA_c711040b
Case setup: Skipped for experiment C48_S2SWA_gefs_c711040b
Case setup: Completed for experiment C48_S2SW_c711040b
Case setup: Completed for experiment C96_atm3DVar_extended_c711040b
Case setup: Skipped for experiment C96_atm3DVar_c711040b
Case setup: Skipped for experiment C96_atmaerosnowDA_c711040b
Case setup: Completed for experiment C96C48_hybatmDA_c711040b
Case setup: Skipped for experiment C96C48_ufs_hybatmDA_c711040b

[emcbot]

Experiment C48_S2SWA_gefs FAILED on Hercules in /work2/noaa/stmp/CI/HERCULES/2614/RUNTESTS/C48_S2SWA_gefs_c711040b

[emcbot]

Experiment C96_atm3DVar FAILED on Hercules in /work2/noaa/stmp/CI/HERCULES/2614/RUNTESTS/C96_atm3DVar_c711040b

[emcbot]

Experiment C48_S2SW FAILED on Hercules in /work2/noaa/stmp/CI/HERCULES/2614/RUNTESTS/C48_S2SW_c711040b

[emcbot]

Experiment C48_ATM FAILED on Hercules in /work2/noaa/stmp/CI/HERCULES/2614/RUNTESTS/C48_ATM_c711040b

[emcbot]

Experiment C96C48_hybatmDA FAILED on Hercules in /work2/noaa/stmp/CI/HERCULES/2614/RUNTESTS/C96C48_hybatmDA_c711040b

[emcbot]

CI Update on Wcoss2 at 05/23/24 03:44:10 PM
=================================================
PR:2614 Reset to Wcoss2-Ready by user and is now restarting CI tests
Canceled all jobs for experiment C96_atm3DVar_extended_c711040b in PR:2614 on Wcoss2
Canceled all jobs for experiment C48_ATM_c711040b in PR:2614 on Wcoss2
Canceled all jobs for experiment C96C48_hybatmDA_c711040b in PR:2614 on Wcoss2
Canceled all jobs for experiment C48_S2SW_c711040b in PR:2614 on Wcoss2

[emcbot]

Experiment C48_S2SWA_gefs FAILED on Orion in /work2/noaa/stmp/CI/ORION/2614/RUNTESTS/C48_S2SWA_gefs_c711040b

[emcbot]

Experiment C48_S2SW FAILED on Orion in /work2/noaa/stmp/CI/ORION/2614/RUNTESTS/C48_S2SW_c711040b

[emcbot]

Experiment C96_atm3DVar FAILED on Orion in /work2/noaa/stmp/CI/ORION/2614/RUNTESTS/C96_atm3DVar_c711040b

[emcbot]

Experiment C48_ATM FAILED on Orion in /work2/noaa/stmp/CI/ORION/2614/RUNTESTS/C48_ATM_c711040b

[emcbot]

Experiment C96C48_hybatmDA FAILED on Orion in /work2/noaa/stmp/CI/ORION/2614/RUNTESTS/C96C48_hybatmDA_c711040b

[emcbot]

CI Update on Wcoss2 at 05/23/24 03:44:39 PM
============================================
Cloning and Building global-workflow PR: 2614
with PID: 106631 on host: clogin05

[emcbot]

Automated global-workflow Testing Results:

Machine: Wcoss2
Start: Thu May 23 15:58:40 UTC 2024 on clogin05
---------------------------------------------------
Build: Completed at 05/23/24 04:09:47 PM
Case setup: Completed for experiment C48_ATM_ac33cccb
Case setup: Skipped for experiment C48mx500_3DVarAOWCDA_ac33cccb
Case setup: Skipped for experiment C48_S2SWA_gefs_ac33cccb
Case setup: Completed for experiment C48_S2SW_ac33cccb
Case setup: Completed for experiment C96_atm3DVar_extended_ac33cccb
Case setup: Skipped for experiment C96_atm3DVar_ac33cccb
Case setup: Skipped for experiment C96_atmaerosnowDA_ac33cccb
Case setup: Completed for experiment C96C48_hybatmDA_ac33cccb
Case setup: Skipped for experiment C96C48_ufs_hybatmDA_ac33cccb

[emcbot]

CI Passed Orion at
Built and ran in directory /work2/noaa/stmp/CI/ORION/2614

[emcbot]

CI Passed Hera at
Built and ran in directory /scratch1/NCEPDEV/global/CI/2614

[emcbot]

CI Passed Hercules at
Built and ran in directory /work2/noaa/stmp/CI/HERCULES/2614

[emcbot]

CI Update on Wcoss2 at 05/23/24 09:52:09 PM
=================================================
PR:2614 Reset to Wcoss2-Ready by user and is now restarting CI tests
Canceled all jobs for experiment C96C48_hybatmDA_ac33cccb in PR:2614 on Wcoss2
Canceled all jobs for experiment C96_atm3DVar_extended_ac33cccb in PR:2614 on Wcoss2
Canceled all jobs for experiment C48_S2SW_ac33cccb in PR:2614 on Wcoss2
Canceled all jobs for experiment C48_ATM_ac33cccb in PR:2614 on Wcoss2

[emcbot]

CI Update on Wcoss2 at 05/23/24 09:52:42 PM
============================================
Cloning and Building global-workflow PR: 2614
with PID: 157329 on host: clogin01

[emcbot]

Automated global-workflow Testing Results:

Machine: Wcoss2
Start: Thu May 23 21:59:37 UTC 2024 on clogin01
---------------------------------------------------
Build: Completed at 05/23/24 10:23:36 PM
Case setup: Completed for experiment C48_ATM_a46a83a0
Case setup: Skipped for experiment C48mx500_3DVarAOWCDA_a46a83a0
Case setup: Skipped for experiment C48_S2SWA_gefs_a46a83a0
Case setup: Completed for experiment C48_S2SW_a46a83a0
Case setup: Completed for experiment C96_atm3DVar_extended_a46a83a0
Case setup: Skipped for experiment C96_atm3DVar_a46a83a0
Case setup: Skipped for experiment C96_atmaerosnowDA_a46a83a0
Case setup: Completed for experiment C96C48_hybatmDA_a46a83a0
Case setup: Skipped for experiment C96C48_ufs_hybatmDA_a46a83a0

[emcbot]

Experiment C48_ATM_a46a83a0 SUCCESS on Wcoss2 at 05/23/24 11:42:10 PM

[emcbot]

Experiment C48_S2SW_a46a83a0 SUCCESS on Wcoss2 at 05/23/24 11:42:13 PM

[emcbot]

Experiment C96C48_hybatmDA_a46a83a0 SUCCESS on Wcoss2 at 05/24/24 12:42:22 AM