MarcelCaron-NOAA commented 1 month ago

Note to developers: You must use this PR template!

Description of Changes

Please include a summary of the changes and the related GitHub issue(s). Please also include relevant motivation and context.

This PR includes most remaining fixes needed for EVSv2-cam:

Previously, CAM precip jobs often failed due to missing MRMS observations, preventing jobs from building combined observed precip files for verification. This PR includes a logic change in which several MRMS accumulations are copied to the prep COMOUT space, and, before computing combined precip observations with sets of MRMS files, each set of files is checked from largest accumulation to smallest until a complete set is found. This fix should both reduce computation load and increase flexibility when some MRMS data are missing.
CAM precip, grid2obs, and snowfall jobs previously ran METplus when needed files were missing, causing errors. In this PR, METplus no longer runs in such cases.
split_by_subset is employed in jobs that use prepbufr data, following norms of other EVS jobs
CAPE timeseries graphics exposed a bug in CAPE stats configurations where models were often being verified against slightly different sets of point observations. This PR fixes that bug.
CAM mail messages were previously submitted in excess. This PR improves the mail message logic to prevent this.
Evaluation period presets that are used to configure plotting processes (e.g., "last31days") previously ended on VDATE-1 day. These periods were changed to end on VDATE.
Jobs were writing subdirectories to COMOUT that were not needed and were left empty. Those unecessary directories are no longer being written.

Developer Questions and Checklist

Is this a high priorty PR? If so, why and is there a date it needs to be merged by?

No. Please test after [#565] is merged.

Do you have any planned upcoming annual leave/PTO?

No.

Are there any changes needed for when the jobs are supposed to run?

No.

[x] The code changes follow NCO's EE2 Standards.
[x] Developer's name is removed throughout the code and have used ${USER} where necessary throughout the code.
[x] References the feature branch for HOMEevs are removed from the code.
[x] J-Job environment variables, COMIN and COMOUT directories, and output follow what has been defined for EVS.
[x] Jobs over 15 minutes in runtime have restart capability.
[x] If applicable, changes in the dev/drivers/scripts or dev/modulefiles have been made in the corresponding ecf/scripts and ecf/defs/evs-nco.def?
[x] Jobs contain the approriate file checking and don't run METplus for any missing data.
[x] Code is using METplus wrappers structure and not calling MET executables directly.
[x] Log is free of any ERRORs or WARNINGs.

Testing Instructions

Please include testing instructions for the PR assignee. Include all relevant input datasets needed to run the tests.

TEST 1: Nominal Runs

For testing most of the changes in this PR

(1) Set up jobs

symlink the EVS_fix directory locally as "fix"
In each driver script, edit the following environment variables: HOMEevs - set to your test EVS directory COMIN - set to /lfs/h2/emc/vpppg/noscrub/emc.vpppg/${NET}_beta5/$evs_ver_2d COMOUT - set to your test output directory SENDMAIL - (optional) set to "NO" DATAROOT - (optional) set to your test DATAROOT directory

(2) Running jobs

I recommend testing the following jobs:

jevs_cam_hireswarw_precip_prep
jevs_cam_hireswarwmem2_precip_prep
jevs_cam_hireswfv3_precip_prep
jevs_cam_hrrr_precip_prep
jevs_cam_namnest_precip_prep

jevs_cam_hireswarw_grid2obs_stats
jevs_cam_hireswarwmem2_grid2obs_stats
jevs_cam_hireswfv3_grid2obs_stats
jevs_cam_hrrr_grid2obs_stats
jevs_cam_namnest_grid2obs_stats
jevs_cam_hireswarw_precip_stats
jevs_cam_hireswarwmem2_precip_stats
jevs_cam_hireswfv3_precip_stats
jevs_cam_hrrr_precip_stats
jevs_cam_namnest_precip_stats

jevs_cam_headline_plots
jevs_cam_grid2obs_last31days_plots
jevs_cam_snowfall_plots*

*corrected [Total: 5 prep jobs; 10 stats jobs; 3 plots jobs]

All precip stats driver scripts can be submitted using qsub -v vhr=21,VDATE=$(date -d "-3 days" +"%Y%m%d") $driver
All other driver scripts can be submitted using qsub -v vhr=00 $driver
All jobs can be submitted at any time

(3) Checking jobs

Log files should be checked by the developer for the following keywords:

check="FATAL\|WARNING\|error\|Killed\|Cgroup\|argument expected\|No such file\|cannot\|failed\|unexpected\|exceeded"
grep "$check" $logfile

Also, confirm that CAPE OBAR stats match between models

TEST 2: Data Denial Runs

For testing mail messages. Also to make sure METplus does not run

(1) Set up jobs

symlink the EVS_fix directory locally as "fix"
In each driver script, edit the following environment variables: HOMEevs - set to your test EVS directory COMIN - set to a nonexistent directory, e.g. /lfs/h2/emc/vpppg/noscrub/BOGUS/${NET}/$evs_ver_2d COMOUT - set to your test output directory SENDMAIL - set to "YES" if it's not already MAILTO - set to "marcel.caron@noaa.gov" (optional: you may also add your own email) DATAROOT - (optional) set to your test DATAROOT directory
In each driver script, also export the following variables: COMINhiresw - set to a nonexistent directory, e.g. "/lfs/h1/ops/prod/com/hiresw/BOGUS" COMINhrrr - set to a nonexistent directory, e.g. "/lfs/h1/ops/prod/com/hrrr/BOGUS" COMINnam - set to a nonexistent directory, e.g. "/lfs/h1/ops/prod/com/nam/BOGUS" COMINobsproc - set to a nonexistent directory, e.g. "/BOGUS" COMINmrms - set to a nonexistent directory, e.g. "/BOGUS" COMINccpa - set to a nonexistent directory, e.g. "/BOGUS" DCOMINsnow - set to a nonexistent directory, e.g. "/BOGUS" EVSINccpa - set to a nonexistent directory, e.g. "/BOGUS" EVSINmrms - set to a nonexistent directory, e.g. "/BOGUS"

(2) Running jobs

I recommend testing the following jobs:

jevs_cam_hireswarw_grid2obs_stats
jevs_cam_hireswarwmem2_grid2obs_stats
jevs_cam_hireswfv3_grid2obs_stats
jevs_cam_hrrr_grid2obs_stats
jevs_cam_namnest_grid2obs_stats
jevs_cam_hireswarw_precip_stats
jevs_cam_hireswarwmem2_precip_stats
jevs_cam_hireswfv3_precip_stats
jevs_cam_hrrr_precip_stats
jevs_cam_namnest_precip_stats
jevs_cam_hireswarw_snowfall_stats
jevs_cam_hireswarwmem2_snowfall_stats
jevs_cam_hireswfv3_snowfall_stats
jevs_cam_hrrr_snowfall_stats
jevs_cam_namnest_snowfall_stats

[Total: 15 stats jobs]

All driver scripts can be submitted using qsub -v vhr=00 $driver
All jobs can be submitted at any time

(3) Checking jobs

Log files should be checked by the developer for the following keywords:

check="FATAL\|WARNING\|error\|Killed\|Cgroup\|argument expected\|No such file\|cannot\|failed\|unexpected\|exceeded"
grep "$check" $logfile

Note: WARNINGs are expected
Also check the mail messages that are produced
Also ensure METplus does not run

malloryprow commented 1 month ago

TEST 1: Nominal Runs

Running the prep jobs! I submitted all with vhr=00.

COMOUT for all is /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr566/evs/v2.0/prep/cam.

jevs_cam_hireswarw_precip_prep

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr566/EVS/dev/drivers/scripts/prep/cam/jevs_cam_hireswarw_precip_prep.o156398561 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswarw_precip_prep.156398561.cbqs01

jevs_cam_hireswarwmem2_precip_prep

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr566/EVS/dev/drivers/scripts/prep/cam/jevs_cam_hireswarwmem2_precip_prep.o156398571 DATA:

jevs_cam_hireswfv3_precip_prep

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr566/EVS/dev/drivers/scripts/prep/cam/jevs_cam_hireswfv3_precip_prep.o156398587 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswfv3_precip_prep.156398587.cbqs01

jevs_cam_hrrr_precip_prep

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr566/EVS/dev/drivers/scripts/prep/cam/jevs_cam_hrrr_precip_prep.o156398589 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hrrr_precip_prep.156398589.cbqs01

jevs_cam_namnest_precip_prep

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr566/EVS/dev/drivers/scripts/prep/cam/jevs_cam_namnest_precip_prep.o156398591 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_namnest_precip_prep.156398591.cbqs01

MarcelCaron-NOAA commented 1 month ago

@malloryprow there was a typo in my recent edit to cam_util.py! It's fixed and should work now. Do you mind rerunning these jobs?

malloryprow commented 1 month ago

TEST 1: Nominal Runs

Reruns!

jevs_cam_hireswarw_precip_prep

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr566/EVS/dev/drivers/scripts/prep/cam/jevs_cam_hireswarw_precip_prep.o156400854 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswarw_precip_prep.156400854.cbqs01

jevs_cam_hireswarwmem2_precip_prep Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr566/EVS/dev/drivers/scripts/prep/cam/jevs_cam_hireswarwmem2_precip_prep.o156400866 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswarwmem2_precip_prep.156400866.cbqs01

jevs_cam_hireswfv3_precip_prep Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr566/EVS/dev/drivers/scripts/prep/cam/jevs_cam_hireswfv3_precip_prep.o156398587 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswfv3_precip_prep.156398587.cbqs01

jevs_cam_hrrr_precip_prep Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr566/EVS/dev/drivers/scripts/prep/cam/jevs_cam_hrrr_precip_prep.o156400869 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hrrr_precip_prep.156400869.cbqs01

jevs_cam_namnest_precip_prep Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr566/EVS/dev/drivers/scripts/prep/cam/jevs_cam_namnest_precip_prep.o156400889 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_namnest_precip_prep.156400889.cbqs01

MarcelCaron-NOAA commented 1 month ago

👍 @malloryprow test1 prep jobs completed successfully and output are as expected ✔️ jevs_cam_hireswarw_precip_prep ✔️ jevs_cam_hireswarwmem2_precip_prep ✔️ jevs_cam_hireswfv3_precip_prep ✔️ jevs_cam_hrrr_precip_prep ✔️ jevs_cam_namnest_precip_prep

malloryprow commented 1 month ago

Stats testing! (jevs_cam_hrrr_grid2obs_stats still running but I think it should be done soon)

jevs_cam_hireswarw_grid2obs_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr566/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hireswarw_grid2obs_stats.o156471598 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswarw_grid2obs_stats.156471598.cbqs01

jevs_cam_hireswarwmem2_grid2obs_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr566/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hireswarwmem2_grid2obs_stats.o156471615 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswarwmem2_grid2obs_stats.156471615.cbqs01

jevs_cam_hireswfv3_grid2obs_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr566/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hireswfv3_grid2obs_stats.o156471646 DATA: =/lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswfv3_grid2obs_stats.156471646.cbqs01

jevs_cam_hrrr_grid2obs_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr566/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hrrr_grid2obs_stats.o156471651 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hrrr_grid2obs_stats.156471651.cbqs01

jevs_cam_namnest_grid2obs_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr566/EVS/dev/drivers/scripts/stats/cam/jevs_cam_namnest_grid2obs_stats.o156471669 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_namnest_grid2obs_stats.156471669.cbqs01

jevs_cam_hireswarw_precip_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr566/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hireswarw_precip_stats.o156471708 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswarw_precip_stats.156471708.cbqs01

jevs_cam_hireswarwmem2_precip_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr566/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hireswarwmem2_precip_stats.o156471719 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswarwmem2_precip_stats.156471719.cbqs01

jevs_cam_hireswfv3_precip_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr566/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hireswfv3_precip_stats.o156471779 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswfv3_precip_stats.156471779.cbqs01

jevs_cam_hrrr_precip_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr566/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hrrr_precip_stats.o156471802 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hrrr_precip_stats.156471802.cbqs01

jevs_cam_namnest_precip_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr566/EVS/dev/drivers/scripts/stats/cam/jevs_cam_namnest_precip_stats.o156471814 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_namnest_precip_stats.156471814.cbqs01

malloryprow commented 1 month ago

Can you do a sync fork when you get the chance?

MarcelCaron-NOAA commented 1 month ago

The following jobs ran cleanly and to completion, except produced warnings for deprecated METplus variable names: 🏁 jevs_cam_hireswarw_grid2obs_stats 🏁 jevs_cam_hireswarwmem2_grid2obs_stats 🏁 jevs_cam_hireswfv3_grid2obs_stats 🏁 jevs_cam_hireswarw_precip_stats 🏁 jevs_cam_hireswarwmem2_precip_stats 🏁 jevs_cam_hireswfv3_precip_stats 🏁 jevs_cam_hrrr_precip_stats 🏁 jevs_cam_namnest_precip_stats

The following ran similarly, but were cut off near the end because of walltime exceedences: 🏁 jevs_cam_hrrr_grid2obs_stats 🏁 jevs_cam_namnest_grid2obs_stats

Can you rerun the jobs with the adjustments I made? I expect the rerun to complete cleanly based on how they ran this time.

MarcelCaron-NOAA commented 1 month ago

@malloryprow also, I figure I'll add in the change to the dates used to represent last31days and last90days (Perry brought this up in a recent email chain). It's a minor change, so I don't think we would need to test all the plots jobs. I'll suggest that we only test the headline plots job. Let me know if you'd prefer to keep that change out of this PR.

malloryprow commented 1 month ago

Yeah! You can put those changes in this PR. We can test the headline plots and maybe a few other random ones, like 31 days and 90 days plots.

MarcelCaron-NOAA commented 1 month ago

Okay! I pushed the change and updated the instructions to include 3 plots jobs for testing that change.

malloryprow commented 1 month ago

Thanks! Do any of the prep jobs need to be re-run? Or good to continue from the stats?

MarcelCaron-NOAA commented 1 month ago

prep jobs are all set! No changes were made to them since testing them. We can continue with stats

malloryprow commented 1 month ago

Thanks! Also I went to move /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr566/evs/v2.0/stats/cam when I noticed a lot of directories in there for VDATEs that jobs were not ran for (like 20240922, 20240923, 20240925, 20240927). Are these supposed to all be there?

MarcelCaron-NOAA commented 1 month ago

No they are not ... they are being written by a few lines of code that loop over all initialization days (not desired). I've made a fix and am testing it now. When it looks right I'll push it. Thanks for catching that

MarcelCaron-NOAA commented 1 month ago

@malloryprow Okay fixed. The stats jobs should only be writing the COMOUT subdirectories with VDATE. Should be good to test now.

AliciaBentley-NOAA commented 1 month ago

Thanks for making that fix, @MarcelCaron-NOAA! Are we up to date with develop? Do we need to do a sync fork on this PR?

MarcelCaron-NOAA commented 1 month ago

@AliciaBentley-NOAA done! Thanks.

malloryprow commented 1 month ago

I can start back with testing this PR today. The prep jobs testing was complete, but I believe we need to continue with the stats testing.

MarcelCaron-NOAA commented 1 month ago

@malloryprow That's right. prep is done and we started testing the stats jobs for the nominal runs, but those jobs need to be rerun.

malloryprow commented 1 month ago

TEST 1: Nominal Runs - stats

Realized I had the directory for this PR named wrong; it was pr566 prior but this PR number is 567, so it have been renamed.

All have COMOUT=/lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/evs/v2.0/stats/cam

jevs_cam_hireswarw_grid2obs_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hireswarw_grid2obs_stats.o157160694 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswarw_grid2obs_stats.157160694.cbqs01

jevs_cam_hireswarwmem2_grid2obs_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hireswarwmem2_grid2obs_stats.o157160698 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswarwmem2_grid2obs_stats.157160698.cbqs01

jevs_cam_hireswfv3_grid2obs_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hireswfv3_grid2obs_stats.o157160729 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswfv3_grid2obs_stats.157160729.cbqs01

jevs_cam_hrrr_grid2obs_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hrrr_grid2obs_stats.o157160807 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hrrr_grid2obs_stats.157160807.cbqs01

jevs_cam_namnest_grid2obs_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_namnest_grid2obs_stats.o157160814 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_namnest_grid2obs_stats.157160814.cbqs01

jevs_cam_hireswarw_precip_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hireswarw_precip_stats.o157160969 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswarw_precip_stats.157160969.cbqs01

jevs_cam_hireswarwmem2_precip_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hireswarwmem2_precip_stats.o157160970 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswarwmem2_precip_stats.157160970.cbqs01

jevs_cam_hireswfv3_precip_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hireswfv3_precip_stats.o157160971 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswfv3_precip_stats.157160971.cbqs01

jevs_cam_hrrr_precip_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hrrr_precip_stats.o157160982 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hrrr_precip_stats.157160982.cbqs01

jevs_cam_namnest_precip_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_namnest_precip_stats.o157160988 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_namnest_precip_stats.157160988.cbqs01

MarcelCaron-NOAA commented 1 month ago

I missed one deprecated variable, but added the change in now. It's a minor change so I don't think a rerun is needed? Maybe we can just rerun jevs_cam_hireswarw_grid2obs_stats, which is relatively quick, to confirm. 🏁 jevs_cam_hireswarw_grid2obs_stats 🏁 jevs_cam_hireswarwmem2_grid2obs_stats 🏁 jevs_cam_hireswfv3

The following completed cleanly and output look normal: ✔️ jevs_cam_hireswarw_precip_stats ✔️ jevs_cam_hireswarwmem2_precip_stats ✔️ jevs_cam_hireswfv3_precip_stats

Still running: ⏳ jevs_cam_hrrr_grid2obs_stats ⏳ jevs_cam_namnest_grid2obs_stats ⏳ jevs_cam_hrrr_precip_stats ⏳ jevs_cam_namnest_precip_stats

malloryprow commented 1 month ago

jevs_cam_hireswarw_grid2obs_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hireswarw_grid2obs_stats.o157165996 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswarw_grid2obs_stats.157165996.cbqs01

MarcelCaron-NOAA commented 1 month ago

Success! Thanks.

The following jobs completed cleanly (or concerns addressed by latest jevs_cam_hireswarw_grid2obs_stats run) and output look normal: ✔️ jevs_cam_hireswarw_grid2obs_stats ✔️ jevs_cam_hireswarwmem2_grid2obs_stats ✔️ jevs_cam_hireswfv3_grid2obs_stats ✔️ jevs_cam_hrrr_precip_stats ✔️ jevs_cam_namnest_precip_stats

Still running: ⏳ jevs_cam_hrrr_grid2obs_stats ⏳ jevs_cam_namnest_grid2obs_stats

MarcelCaron-NOAA commented 1 month ago

The following completed cleanly and output look normal: ✔️ jevs_cam_namnest_grid2obs_stats ✔️ jevs_cam_hrrr_grid2obs_stats

👍 All of the test1 stats jobs ran successfully. No further concerns from me. (Edit to add that I confirmed CAPE fix is working)

malloryprow commented 1 month ago

TEST 1: Nominal Runs - plots

jevs_cam_headline_plots

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/plots/cam/jevs_cam_headline_plots.o157174773 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_headline_plots.157174773.cbqs01

jevs_cam_grid2obs_last31days_plots

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/plots/cam/jevs_cam_grid2obs_plots_last31days.o157174875 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_grid2obs_plots_last31days.157174875.cbqs01

jevs_cam_snowfall_plots

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/plots/cam/jevs_cam_snowfall_plots.o157175658 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_snowfall_plots.157175658.cbqs01

MarcelCaron-NOAA commented 1 month ago

The following completed cleanly, output looks good, in particular images are ending on the valid date: ✔️ jevs_cam_grid2obs_last31days_plots ✔️ jevs_cam_snowfall_plots ✔️ jevs_cam_headline_plots

👍 All jobs for test1 have completed successfully, no more concerns from me about those

malloryprow commented 1 month ago

I'll update the scripts for Test 2! I might save the running of them until Monday, just in case those jobs don't get done until late and the DATA directories gets scrubbed from stmp over the weekend.

MarcelCaron-NOAA commented 1 month ago

I'll update the scripts for Test 2! I might save the running of them until Monday, just in case those jobs don't get done until late and the DATA directories gets scrubbed from stmp over the weekend.

Thanks @malloryprow. I support running these Monday!

malloryprow commented 1 month ago

I'm setting the cam stats scripts and I'm noticing for some of MAILTO some have @AndrewBenjamin-NOAA on them and some don't (it is just you @MarcelCaron-NOAA).

Can we add @AndrewBenjamin-NOAA to the MAILTO jobs he is missing on?

MarcelCaron-NOAA commented 1 month ago

@malloryprow I'm not seeing any instances in my local repository. Do you mind listing the ones you found?

malloryprow commented 1 month ago

Ah, I may have got some things crossed then. If you aren't seeing it then, I trust things are good. I was out all this week but I'll be picking this up again this afternoon I hope!

malloryprow commented 1 month ago

TEST 2: Data Denial Runs

jevs_cam_hireswarw_grid2obs_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hireswarw_grid2obs_stats.o157801293 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswarw_grid2obs_stats.157801293.cbqs01

jevs_cam_hireswarwmem2_grid2obs_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hireswarwmem2_grid2obs_stats.o157801770 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswarwmem2_grid2obs_stats.157801770.cbqs01

jevs_cam_hireswfv3_grid2obs_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hireswfv3_grid2obs_stats.o157801894 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswfv3_grid2obs_stats.157801894.cbqs01

jevs_cam_hrrr_grid2obs_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hrrr_grid2obs_stats.o157801995 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hrrr_grid2obs_stats.157801995.cbqs01

jevs_cam_namnest_grid2obs_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_namnest_grid2obs_stats.o157802194 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_namnest_grid2obs_stats.157802194.cbqs01

jevs_cam_hireswarw_precip_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hireswarw_precip_stats.o157802323 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswarw_precip_stats.157802323.cbqs01

jevs_cam_hireswarwmem2_precip_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hireswarwmem2_precip_stats.o157802657 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswarwmem2_precip_stats.157802657.cbqs01

jevs_cam_hireswfv3_precip_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hireswfv3_precip_stats.o157802715 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswfv3_precip_stats.157802715.cbqs01

jevs_cam_hrrr_precip_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hrrr_precip_stats.o157803017 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hrrr_precip_stats.157803017.cbqs01

jevs_cam_namnest_precip_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_namnest_precip_stats.o157803083 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_namnest_precip_stats.157803083.cbqs01

jevs_cam_hireswarw_snowfall_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hireswarw_snowfall_stats.o157803549 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswarw_snowfall_stats.157803549.cbqs01

jevs_cam_hireswarwmem2_snowfall_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hireswarwmem2_snowfall_stats.o157803663 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswarwmem2_snowfall_stats.157803663.cbqs01

jevs_cam_hireswfv3_snowfall_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hireswfv3_snowfall_stats.o157803837 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswfv3_snowfall_stats.157803837.cbqs01

jevs_cam_hrrr_snowfall_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hrrr_snowfall_stats.o157804042 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hrrr_snowfall_stats.157804042.cbqs01

jevs_cam_namnest_snowfall_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_namnest_snowfall_stats.o157804351 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_namnest_snowfall_stats.157804351.cbqs01

MarcelCaron-NOAA commented 1 month ago

@malloryprow Thanks for the runs! All completed but none ran as expected so I need to make and test some fixes. I'll update again soon.

METplus ran when it shouldn't have: ❌ jevs_cam_hireswarw_grid2obs_stats ❌ jevs_cam_hireswarwmem2_grid2obs_stats ❌ jevs_cam_hireswfv3_grid2obs_stats ❌ jevs_cam_hireswarw_snowfall_stats ❌ jevs_cam_hireswarwmem2_snowfall_stats ❌ jevs_cam_hireswfv3_snowfall_stats ❌ jevs_cam_hrrr_snowfall_stats ❌ jevs_cam_namnest_snowfall_stats

No mail messages: ❌ jevs_cam_hrrr_grid2obs_stats ❌ jevs_cam_namnest_grid2obs_stats ❌ jevs_cam_hireswarw_precip_stats ❌ jevs_cam_hireswarwmem2_precip_stats ❌ jevs_cam_hireswfv3_precip_stats ❌ jevs_cam_hrrr_precip_stats ❌ jevs_cam_namnest_precip_stats

malloryprow commented 1 month ago

Okay! Let me know if something looks wrong in my set-up. I can re-run if needed.

MarcelCaron-NOAA commented 1 month ago

No, the setups look good, it's my fault. I am just testing a fix addressing the snowfall runs and will push it soon.

I also forgot to add the following bogus directories to the PR instructions (edited just now). Can you add them to the 15 driver scripts?

(something like the following should work):

export COMINobsproc="/BOGUS"
export COMINmrms="/BOGUS"
export COMINccpa="/BOGUS"
export DCOMINsnow="/BOGUS"
export EVSINccpa="/BOGUS"
export EVSINmrms="/BOGUS"

The fix and the added dev variables should address all the issues!

malloryprow commented 1 month ago

Yeah, I'll do that! I see the snowfall just got pushed. Good to try again?

MarcelCaron-NOAA commented 1 month ago

Not yet, sorry! I'm still testing the regular snowfall job as a final check. Should be done soon!

MarcelCaron-NOAA commented 1 month ago

@malloryprow Okay with those last commits, we are good to go! Thanks

malloryprow commented 1 month ago

TEST 2: Data Denial Runs - Try 2

jevs_cam_hireswarw_grid2obs_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hireswarw_grid2obs_stats.o157829124 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswarw_grid2obs_stats.157829124.cbqs01

jevs_cam_hireswarwmem2_grid2obs_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hireswarwmem2_grid2obs_stats.o157829151 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswarwmem2_grid2obs_stats.157829151.cbqs01

jevs_cam_hireswfv3_grid2obs_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hireswfv3_grid2obs_stats.o157829174 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswfv3_grid2obs_stats.157829174.cbqs01

jevs_cam_hrrr_grid2obs_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hrrr_grid2obs_stats.o157829206 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hrrr_grid2obs_stats.157829206.cbqs01

jevs_cam_namnest_grid2obs_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_namnest_grid2obs_stats.o157829214 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_namnest_grid2obs_stats.157829214.cbqs01

jevs_cam_hireswarw_precip_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hireswarw_precip_stats.o157829458 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswarw_precip_stats.157829458.cbqs01

jevs_cam_hireswarwmem2_precip_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hireswarwmem2_precip_stats.o157829498 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswarwmem2_precip_stats.157829498.cbqs01

jevs_cam_hireswfv3_precip_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hireswfv3_precip_stats.o157829530 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswfv3_precip_stats.157829530.cbqs01

jevs_cam_hrrr_precip_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hrrr_precip_stats.o157829565 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hrrr_precip_stats.157829565.cbqs01

jevs_cam_namnest_precip_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_namnest_precip_stats.o157829703 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_namnest_precip_stats.157829703.cbqs01

jevs_cam_hireswarw_snowfall_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hireswarw_snowfall_stats.o157829753 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswarw_snowfall_stats.157829753.cbqs01

jevs_cam_hireswarwmem2_snowfall_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hireswarwmem2_snowfall_stats.o157829829 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswarwmem2_snowfall_stats.157829829.cbqs01

jevs_cam_hireswfv3_snowfall_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hireswfv3_snowfall_stats.o157829877 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswfv3_snowfall_stats.157829877.cbqs01

jevs_cam_hrrr_snowfall_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hrrr_snowfall_stats.o157829940 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hrrr_snowfall_stats.157829940.cbqs01

jevs_cam_namnest_snowfall_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_namnest_snowfall_stats.o157804351 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_namnest_snowfall_stats.157829964.cbqs01

MarcelCaron-NOAA commented 1 month ago

@malloryprow All of the following jobs completed with mail messages and METplus did not run: ✔️ jevs_cam_hireswarw_precip_stats ✔️ jevs_cam_hireswarwmem2_precip_stats ✔️ jevs_cam_hireswfv3_precip_stats ✔️ jevs_cam_hrrr_precip_stats ✔️ jevs_cam_namnest_precip_stats ✔️ jevs_cam_hireswarw_snowfall_stats ✔️ jevs_cam_hireswarwmem2_snowfall_stats ✔️ jevs_cam_hireswarwfv3_snowfall_stats ✔️ jevs_cam_hrrr_snowfall_stats ✔️ jevs_cam_namnest_snowfall_stats

The following had an error I missed (should only be warnings). Sorry for the back and forth on this. I will have a fix in place tomorrow and will update this thread when I've tested it successfully! ❌ jevs_cam_hireswarw_grid2obs_stats ❌ jevs_cam_hireswarwmem2_grid2obs_stats ❌ jevs_cam_hireswfv3_grid2obs_stats ❌ jevs_cam_hrrr_grid2obs_stats ❌ jevs_cam_namnest_grid2obs_stats

AliciaBentley-NOAA commented 1 month ago

Thanks, @MarcelCaron-NOAA! I think this is a really good PR that will help streamline CAM a lot!

MarcelCaron-NOAA commented 1 month ago

@AliciaBentley-NOAA I think so! This code should be much cleaner.

MarcelCaron-NOAA commented 1 month ago

@malloryprow Alright the recent pushes were tested and should fix all of the remaining issues with the grid2obs stats data denial runs. We're good to continue testing!

malloryprow commented 1 month ago

TEST 2: Data Denial Runs - Try 3

COMOUT is /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/evs/v2.0/stats/cam.

jevs_cam_hireswarw_grid2obs_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/jevs_cam_hireswarw_grid2obs_stats.o158275745 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswarw_grid2obs_stats.158275745.cbqs01

jevs_cam_hireswarwmem2_grid2obs_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/jevs_cam_hireswarwmem2_grid2obs_stats.o158276283 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswarwmem2_grid2obs_stats.158276283.cbqs01

jevs_cam_hireswfv3_grid2obs_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/jevs_cam_hireswfv3_grid2obs_stats.o158276308 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hireswfv3_grid2obs_stats.158276308.cbqs01

jevs_cam_hrrr_grid2obs_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/jevs_cam_hrrr_grid2obs_stats.o158276315 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hrrr_grid2obs_stats.158276315.cbqs01

jevs_cam_namnest_grid2obs_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/jevs_cam_namnest_grid2obs_stats.o158276349 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_namnest_grid2obs_stats.158276349.cbqs01

MarcelCaron-NOAA commented 1 month ago

The following completed cleanly and submitted mail messages for missing files: ✔️ jevs_cam_hireswarw_grid2obs_stats ✔️ jevs_cam_hireswarwmem2_grid2obs_stats ✔️ jevs_cam_hireswfv3_grid2obs_stats

The following exited with an error: ❌ jevs_cam_hrrr_grid2obs_stats ❌ jevs_cam_namnest_grid2obs_stats

These two jobs need a minor fix related to printing messages for missing firewxnest files. Can you rerun just the two with the fix I just added?

malloryprow commented 1 month ago

TEST 2: Data Denial Runs - Try 4

COMOUT is /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/evs/v2.0/stats/cam (moved previous one to cam.try23).

jevs_cam_hrrr_grid2obs_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam/jevs_cam_hrrr_grid2obs_stats.o158280359 DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_hrrr_grid2obs_stats.158280359.cbqs01

jevs_cam_namnest_grid2obs_stats

Log File: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr567/EVS/dev/drivers/scripts/stats/cam DATA: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_namnest_grid2obs_stats.158280374.cbqs01

MarcelCaron-NOAA commented 1 month ago

@malloryprow Thank you!

The following completed cleanly and submitted mail messages successfully: ✔️ jevs_cam_hrrr_grid2obs_stats ✔️ jevs_cam_namnest_grid2obs_stats

👍 That concludes all runs for TEST 2. No more concerns from me about the jobs affected by this PR!

AndrewBenjamin-NOAA commented 1 month ago

Thanks Marcel! I will take a look at the changed files and submit my review.

malloryprow commented 1 month ago

Thanks for your work @MarcelCaron-NOAA! Be sure to cross off anything from the Fixes and Additions this covers!

NOAA-EMC / EVS

Bugfix/cam fixes 2 #567

Description of Changes

Developer Questions and Checklist

Testing Instructions

TEST 1: Nominal Runs

(1) Set up jobs

(2) Running jobs

(3) Checking jobs

TEST 2: Data Denial Runs

(1) Set up jobs

(2) Running jobs

(3) Checking jobs

TEST 1: Nominal Runs

jevs_cam_hireswarw_precip_prep

jevs_cam_hireswarwmem2_precip_prep

jevs_cam_hireswfv3_precip_prep

jevs_cam_hrrr_precip_prep

jevs_cam_namnest_precip_prep

TEST 1: Nominal Runs

jevs_cam_hireswarw_precip_prep

jevs_cam_hireswarw_grid2obs_stats

jevs_cam_hireswarwmem2_grid2obs_stats

jevs_cam_hireswfv3_grid2obs_stats

jevs_cam_hrrr_grid2obs_stats

jevs_cam_namnest_grid2obs_stats

jevs_cam_hireswarw_precip_stats

jevs_cam_hireswarwmem2_precip_stats

jevs_cam_hireswfv3_precip_stats

jevs_cam_hrrr_precip_stats

jevs_cam_namnest_precip_stats

TEST 1: Nominal Runs - stats

jevs_cam_hireswarw_grid2obs_stats

jevs_cam_hireswarwmem2_grid2obs_stats

jevs_cam_hireswfv3_grid2obs_stats

jevs_cam_hrrr_grid2obs_stats

jevs_cam_namnest_grid2obs_stats

jevs_cam_hireswarw_precip_stats

jevs_cam_hireswarwmem2_precip_stats

jevs_cam_hireswfv3_precip_stats

jevs_cam_hrrr_precip_stats

jevs_cam_namnest_precip_stats

jevs_cam_hireswarw_grid2obs_stats

TEST 1: Nominal Runs - plots

jevs_cam_headline_plots

jevs_cam_grid2obs_last31days_plots

jevs_cam_snowfall_plots

TEST 2: Data Denial Runs

jevs_cam_hireswarw_grid2obs_stats

jevs_cam_hireswarwmem2_grid2obs_stats

jevs_cam_hireswfv3_grid2obs_stats

jevs_cam_hrrr_grid2obs_stats

jevs_cam_namnest_grid2obs_stats

jevs_cam_hireswarw_precip_stats

jevs_cam_hireswarwmem2_precip_stats

jevs_cam_hireswfv3_precip_stats

jevs_cam_hrrr_precip_stats

jevs_cam_namnest_precip_stats

jevs_cam_hireswarw_snowfall_stats

jevs_cam_hireswarwmem2_snowfall_stats

jevs_cam_hireswfv3_snowfall_stats

jevs_cam_hrrr_snowfall_stats

jevs_cam_namnest_snowfall_stats

TEST 2: Data Denial Runs - Try 2

jevs_cam_hireswarw_grid2obs_stats

jevs_cam_hireswarwmem2_grid2obs_stats

jevs_cam_hireswfv3_grid2obs_stats

jevs_cam_hrrr_grid2obs_stats

jevs_cam_namnest_grid2obs_stats

jevs_cam_hireswarw_precip_stats

jevs_cam_hireswarwmem2_precip_stats

jevs_cam_hireswfv3_precip_stats

jevs_cam_hrrr_precip_stats

jevs_cam_namnest_precip_stats

jevs_cam_hireswarw_snowfall_stats

jevs_cam_hireswarwmem2_snowfall_stats

jevs_cam_hireswfv3_snowfall_stats

jevs_cam_hrrr_snowfall_stats

jevs_cam_namnest_snowfall_stats

TEST 2: Data Denial Runs - Try 3