search

ufs-community / ufs-srweather-app

UFS Short-Range Weather Application
Other
55 stars 116 forks source link

Malformed rocoto xml created if running <6 hr forecast with snowfall verification #996

Open mkavulich opened 8 months ago

mkavulich commented 8 months ago

Expected behavior

NOHRSC snowfall observations are only available in 6-hour accumulation increments. There should be some kind of check in place to ensure that the user is not attempting to run a forecast for a shorter time if they select "ASNOW" as one of the verification variables.

Current behavior

Due to a oversight in the addition of snowfall verification, there is no check to ensure we have valid observation times when attempting to verify against NOHRSC snowfall observations. So if you run a forecast with no valid observation times, a malformed Rocoto XML file will be created.

Machines affected

All

Steps To Reproduce

The easiest way to reproduce this is by modifying the custom_ESGgrid_Great_Lakes_snow_8km test and running it on Hera or Jet

  1. edit ufs-srweather-app/tests/WE2E/test_configs/custom_grids/config.custom_ESGgrid_Great_Lakes_snow_8km.yaml and change FCST_LEN_HRS to 3 (or some other value less than 6)
  2. Run the test
  3. See the bug: 
    
$ ./run_WE2E_tests.py -m=hera -a fv3lam -q -t custom_ESGgrid_Great_Lakes_snow_8km
Checking that all tests are valid
Will run 1 tests:
/scratch1/BMC/hmtb/kavulich/UFS/workdir/test_develop/2023-12-15/intel/ufs-srweather-app/tests/WE2E/test_configs/custom_grids/config.custom_ESGgrid_Great_Lakes_snow_8km.yaml
Calling workflow generation function for test custom_ESGgrid_Great_Lakes_snow_8km

Workflow for test custom_ESGgrid_Great_Lakes_snow_8km successfully generated in /scratch1/BMC/hmtb/kavulich/UFS/workdir/test_develop/2023-12-15/intel/expt_dirs/custom_ESGgrid_Great_Lakes_snow_8km

All experiments have been generated; Experiment file WE2E_tests_20231219173951.yaml created Writing information for all experiments to WE2E_tests_20231219173951.yaml Checking tests available for monitoring... Starting experiment custom_ESGgrid_Great_Lakes_snow_8km running 12/19/23 17:39:55 UTC :: FV3LAM_wflow.xml :: ERROR: No tag or values specified in one or more metatasks 12/19/23 17:39:59 UTC :: FV3LAM_wflow.xml :: ERROR: No tag or values specified in one or more metatasks Setup complete; monitoring 1 experiments Use ctrl-c to pause job submission/monitoring 12/19/23 17:40:00 UTC :: FV3LAM_wflow.xml :: ERROR: No tag or values specified in one or more metatasks 12/19/23 17:40:01 UTC :: FV3LAM_wflow.xml :: ERROR: No tag or values specified in one or more metatasks WARNING:Tasks have not yet been submitted for experiment custom_ESGgrid_Great_Lakes_snow_8km; it could be that your jobs are being throttled at the system level.

            If you continue to see this message, there may be an error with your
            experiment configuration, such as an incorrect queue or account number.

            You can use ctrl-c to pause this script and inspect log files.

This loop will continue forever because no tasks will submit due to the malformed `FV3LAM_wflow.xml` file. Specifically, all the snow verification tasks such as `PcpCombine_fcst_ASNOW_all_accums_all_mems` will be set with a variable "ACCUM_HH" which is empty.

## Detailed Description of Fix
We will need to implement some check that the forecast is at least 6 hours if ASNOW is selected as one of the `VX_FIELDS`, and either quit at the workflow generation step with a descriptive error message, or remove `ASNOW` from `VX_FIELDS` and continue with workflow generation.