Adding split_by_subset to firewxnest

PerryShafran-NOAA commented 1 month ago

Note to developers: You must use this PR template!

Description of Changes

Please include a summary of the changes and the related GitHub issue(s). Please also include relevant motivation and context.

Adding split_by_subset processing to the firewxnest stats job. This reduced the walltime from 6 minutes to 5 minutes.

Developer Questions and Checklist

Is this a high priorty PR? If so, why and is there a date it needs to be merged by? No
Do you have any planned upcoming annual leave/PTO? Yes, I will be on leave much of the day on Friday (exact times not known yet) Doctor's appointments in the morning of Wednesday, Sep. 4 Otherwise I am here.
Are there any changes needed for when the jobs are supposed to run? No
[x ] The code changes follow NCO's EE2 Standards.
[x ] Developer's name is removed throughout the code and have used ${USER} where necessary throughout the code.
[x ] References the feature branch for HOMEevs are removed from the code.
[ x] J-Job environment variables, COMIN and COMOUT directories, and output follow what has been defined for EVS.
[NA ] Jobs over 15 minutes in runtime have restart capability.
[x ] If applicable, changes in the dev/drivers/scripts or dev/modulefiles have been made in the corresponding ecf/scripts and ecf/defs/evs-nco.def?
[x ] Jobs contain the approriate file checking and don't run METplus for any missing data.
[x ] Code is using METplus wrappers structure and not calling MET executables directly.
[ x] Log is free of any ERRORs or WARNINGs.

Testing Instructions

Please include testing instructions for the PR assignee. Include all relevant input datasets needed to run the tests.

Clone my fork, checkout feature/firewxnest_v2, link to the fix directory. cd EVS/dev/drivers/scripts/stats/cam Run all 24 hours worth of firewxnest stats jobs. Please be sure that when you run vhr=23 that all the jobs for hours 00 to 22 are complete as vhr=23 runs the gather job.

qsub -v vhr=00 jevs_cam_nam_firewxnest_grid2obs_stats.sh

To compare with emc.vpppg, you might want to run for PDYm2 (job regularly set to PDYm1), but even if you run for PDYm1 we can compare the small stats files that are available.

malloryprow commented 1 month ago

I ran jevs_cam_nam_firewxnest_grid2obs_stats.sh for vhr=0-23 with VDATE=20240828.

Log Files: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr540/EVS/dev/drivers/scripts/stats/cam/jevs_cam_nam_firewxnest_grid2obs_stats_00.o15069 DATAs: /lfs/h2/emc/stmp/mallory.row/evs_test/prod/tmp/jevs_cam_nam_firewxnest_grid2obs_stats_00.15069 COMOUT: /lfs/h2/emc/vpppg/noscrub/mallory.row/verification/EVS_PRs/pr540/evs/v2.0/stats/cam

PerryShafran-NOAA commented 1 month ago

@malloryprow Do we know if this WARNING is being resolved by beta5?

WARNING: 
WARNING: VL1L2Info::compute_stats() -> Skipping 35 of 185 vector pairs for which the direction difference is undefined.
WARNING: Set the "wind_thresh" and "wind_logic" configuration options to exclude zero vectors.
WARNING:

malloryprow commented 1 month ago

Yup it will be!

PerryShafran-NOAA commented 1 month ago

Good! I see the warning in the emc.vpppg output file as well.

No other ERRORs or WARNINGs. The final stat file is identical to the stat file in emc.vpppg. Walltimes are slightly less than what we see in emc.vpppg. I think that we're good to go here.

malloryprow commented 1 month ago

@AndrewBenjamin-NOAA Want to give this a quick review so we can merge it in before the holiday weekend?

malloryprow commented 1 month ago

Thanks for the changes, Perry! Cross of items on the Fixes and Additions doc this addresses!

NOAA-EMC / EVS