This PR fixes several bugs discovered in the verification portion of the app during recent testing by the DTC. The bugs are:
1) Change to parm/data_locations.yml to account for the change in prebpufr (NDAS) obs file names on May 22, 2024. This is currently causing get_obs_ndas tasks to fail for cycles at or after this date. [Bug found by @michelleharrold.]
2) Change to parm/wflow/verify_det.yaml so that all tasks have a cycldefs statement by default. This bug was causing GridStat workflow tasks for CCPA and NOHRSC obs to be created for cycles not defined for the workflow (these extraneous cycles probably correspond to the default set of cycles that a task gets assigned by ROCOTO when it does not contain an explicit cycledefs statement). [Bug found by @michelleharrold, solution by @mkavulich.]
3) Change to scripts/exregional_run_met_gridstat_or_pointstat.sh to append a string for the cycle date ("_YYYYMDDHH") to the name of the metplus log file for deterministic GridStat and PointStat tasks. This was causing the metplus log file for GridStat for a given cycle tasks to be overwritten by those for other cycles. [Bug found by @michelleharrold.]
4) Refactoring of scripts/exregional_get_verif_obs.sh to account for multiple overlapping cycles. It was found that when the experiment contains multiple cycles, the get_obs_ccpa, get_obs_mrms, and get_obs_ndas from the different cycles write to the same directories and clobber the output of other cycles or cause crashes (e.g. a directory is not empty when it is expected to be). [Bug found by @michelleharrold and @willmayfield.]. This script is refactored so that:
a) "Raw" directories, i.e. the directories in which the data is pulled by the retrieve_data.py script, for each cycle or observation day are unique (contain either the cycle (raw_cycYYYYMMDDHH) or the date (raw_dayYYYYMMDD) in their names). Whether a cycle-based or obs-day-based raw directory is used depends on the obs type: CCPA raw directories are cycle-based, and MRMS and NDAS raw directories are day-based.
b) There are checks on directories and files to make sure the get_obs_... task for another cycle is not writing to the directory and/or file the current cycle wishes to modify (write to, move, delete, etc).
c) Data pulls and the final set of processed observations are organized so that there are no extra observation files that lie outside the range of the forecasts.
d) Extra documentation is included in the script to describe in detail the behavior of the script for each obs type (CCPA, MRMS, NDAS).
Type of change
[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update
TESTS CONDUCTED:
[x] hera.intel
[ ] orion.intel
[ ] hercules.intel
[ ] cheyenne.intel
[ ] cheyenne.gnu
[ ] derecho.intel
[ ] gaea.intel
[ ] gaeac5.intel
[ ] jet.intel
[ ] wcoss2.intel
[ ] NOAA Cloud (indicate which platform)
[ ] Jenkins
[ ] fundamental test suite
[ ] comprehensive tests (specify which if a subset was used)
DOCUMENTATION:
ISSUE:
CHECKLIST
[ ] My code follows the style guidelines in the Contributor's Guide
[ ] I have performed a self-review of my own code using the Code Reviewer's Guide
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] My changes need updates to the documentation. I have made corresponding changes to the documentation
[ ] My changes do not require updates to the documentation (explain).
[ ] My changes generate no new warnings
[ ] New and existing tests pass with my changes
[ ] Any dependent changes have been merged and published
LABELS (optional):
A Code Manager needs to add the following labels to this PR:
DESCRIPTION OF CHANGES:
This PR fixes several bugs discovered in the verification portion of the app during recent testing by the DTC. The bugs are:
1) Change to
parm/data_locations.ymlto account for the change in prebpufr (NDAS) obs file names on May 22, 2024. This is currently causingget_obs_ndastasks to fail for cycles at or after this date. [Bug found by @michelleharrold.]2) Change to
parm/wflow/verify_det.yamlso that all tasks have acycldefsstatement by default. This bug was causingGridStatworkflow tasks forCCPAandNOHRSCobs to be created for cycles not defined for the workflow (these extraneous cycles probably correspond to the default set of cycles that a task gets assigned byROCOTOwhen it does not contain an explicitcycledefsstatement). [Bug found by @michelleharrold, solution by @mkavulich.] 3) Change toscripts/exregional_run_met_gridstat_or_pointstat.shto append a string for the cycle date ("_YYYYMDDHH") to the name of the metplus log file for deterministicGridStatandPointStattasks. This was causing the metplus log file for GridStat for a given cycle tasks to be overwritten by those for other cycles. [Bug found by @michelleharrold.]4) Refactoring of
scripts/exregional_get_verif_obs.shto account for multiple overlapping cycles. It was found that when the experiment contains multiple cycles, theget_obs_ccpa,get_obs_mrms, andget_obs_ndasfrom the different cycles write to the same directories and clobber the output of other cycles or cause crashes (e.g. a directory is not empty when it is expected to be). [Bug found by @michelleharrold and @willmayfield.]. This script is refactored so that: a) "Raw" directories, i.e. the directories in which the data is pulled by theretrieve_data.pyscript, for each cycle or observation day are unique (contain either the cycle (raw_cycYYYYMMDDHH) or the date (raw_dayYYYYMMDD) in their names). Whether a cycle-based or obs-day-based raw directory is used depends on the obs type: CCPA raw directories are cycle-based, and MRMS and NDAS raw directories are day-based. b) There are checks on directories and files to make sure theget_obs_...task for another cycle is not writing to the directory and/or file the current cycle wishes to modify (write to, move, delete, etc). c) Data pulls and the final set of processed observations are organized so that there are no extra observation files that lie outside the range of the forecasts. d) Extra documentation is included in the script to describe in detail the behavior of the script for each obs type (CCPA, MRMS, NDAS).Type of change
TESTS CONDUCTED:
DOCUMENTATION:
ISSUE:
CHECKLIST
LABELS (optional):
A Code Manager needs to add the following labels to this PR:
CONTRIBUTORS (optional):