Closed natalie-perlin closed 11 months ago
@mkavulich - some input on the changes from #PR-864 and whether they could could have affected this test could be really helpful! I'm not sure the
Some changes that could be relevant:
Verification task names likely changed from get_obs_*
to get_verif_obs
as a part of #PR-864, yet the tasks that fail in MET_ensemble_verification_only_vx_time_lag are named get_obs_ccpa, get_obs_mrms, get_obs_ndas, despite being part of a verification task workflow.
@natalie-perlin -
I was able to clone the develop branch on Orion, build the SRW App, then submit the MET_ensemble_verification_only_vx_time_lag
. My test failed due to HPSS not being available on Orion. The get_obs_*
tasks should be pointing to the JREGIONAL_GET_VERIF_OBS
j-job file, i.e., <command>&LOAD_MODULES_RUN_TASK_FP; "get_obs" "&JOBSdir;/JREGIONAL_GET_VERIF_OBS"</command>
The test was fundamentally changed in PR #864 to require the verification data to be pulled from HPSS (please see lines 31-34 of the MET_ensemble_verification_only_vx_time_lag configuration file). The test no longer uses the staged data. With this change, this test can only be run on Hera and Jet. It should also be noted that the data in question appears to contain restricted data. If you aren't a member of the rstprod project, then you will be unable to pull the necessary data from HPSS, resulting in the test failing.
@natalie-perlin - I can confirm that the removal of line 31-34 in the MET_ensemble_verification_only_vx_time_lag
configuration file will allow the test to run without issue using the staged data. However, as noted above, the purpose of the test is try and pull the verification from HPSS and then run the test.
@MichaelLueken thanks for jumping in with a reply. Your summary is correct: these two WE2E tests are intended to only check for data on HPSS. I used HPSS data for the MET_ensemble_verification_only_vx_time_lag
test because, at the time, data was only staged for that test on Hera, so it couldn't be run on other machines anyway. In addition, the function of the tasks get_obs_ccpa
, get_obs_mrms
, and get_obs_ndas
changed with that PR. They should be run regardless of whether data is being pulled from HPSS or disk (this makes the formatting of config.yaml much easier for most cases); in the latter case the task checks to ensure all the necessary data is available on disk.
If there is a desire to make the time-lag test use staged data that would be fine, but at least one of the verification tests should be run pulling data from HPSS to test that functionality.
@mkavulich @MichaelLueken - thank you for your comments
EXTRN_MDL_DATA_STORES: aws nomads
in hera.yaml@natalie-perlin - As per today's meeting, please ensure that you log into AIM and request access to the rstprod project. You will be asked to provide justification to be granted permission. If you include:
The Short-Range Weather Application (SRW App) runs workflow end-to-end tests to ensure that modifications to the code don't adversely affect development. Among these tests, there are verification tests that require observational data from HPSS. Unfortunately, these data sets are included in tarballs that also contain restricted data. Due to this, verification tests that need to pull data from HPSS are failing due to the lack of rstprod project access.
access should be granted. Once you have access to rstprod on RDHPCS, you will need to let HPSS know that you have been granted access to rstprod on RDHPCS so that you can pull the tarball that contains restricted data from HPSS. The email for the HPSS helpdesk is rdhpcs.hpss.help@noaa.gov. Skylar Nelson is the lead for the HPSS helpdesk, so including an email to him might expedite the process.
Closing this issue now.
MET verification tests use modules met and metplus from software stacks on Tier 1 machines, and the changes were implemented in PR-826 (https://github.com/ufs-community/ufs-srweather-app/pull/826) Since then, changes that were implemented affected the MET verification tasks, and MET_ensemble_verification_only_vx_time_lag no longer seem to work (Tested Hera, Gaea, Orion; new platform Derecho). Tasks get_obs_ccpa, get_obs_mrms, get_obs_ndas fail.
Log files could be viewed on Hera: /scratch1/NCEPDEV/stmp2/Natalie.Perlin/SRW/expt_dirs/MET_ensemble_verification_only_vx_time_lag/log/get_obs_ndas_2021050500.log, get_obs_mrms_2021050500.log, get_obs_ccpa_2021050500.log
Attached are the getobs*_2021050500.log files, var_defns.sh and generated FV3LAM_wflow.xml workflow.
Expected behavior
MET_ensemble_verification_only_vx_time_lag test passes successfully on Hera (intel and gnu, Gaea, Orion, Jet, Derecho). Tasks get_obs_ccpa, get_obs_mrms, get_obs_ndas do not need to be run, as the data is staged on these systems.
Current behavior
Tasks that fail are get_obs_ccpa, get_obs_mrms, get_obs_ndas.
Machines affected
Any system running SRW
Steps To Reproduce
Example for Orion :
See the bug... -->
In MET_ensemble_verification_only_vx_time_lag tests done before merging the PR-826, no get_obs_ccpa, get_obs_mrms, get_obs_ndas were run, as all of the data were staged on each machine.
An example of a successful MET_ensemble_verification_only_vx_time_lag test on Hera: SRW base directory: /scratch1/NCEPDEV/stmp2/Natalie.Perlin/SRW/srw-dev-met Experiment directory: /scratch1/NCEPDEV/stmp2/Natalie.Perlin/SRW/INTEL/MET_ensemble_verification_only_vx_time_lag
Detailed Description of Fix (optional)
May need to be related to configurations in parm/wflow/*.yaml, and ./ush/machine/verify_*.yaml files
Additional Information (optional)
There are differences between machine files used in PR-826, i.e., setting the OBS data directories, and their current versions (and also earlier version, before the PR-826). Example for Hera; used the following data in PR-826:
Current hera.yaml and the machine file before the merge of PR-826 contain the following:
Possible Implementation (optional)
Output (optional)
get_obs_ccpa_2021050500.log get_obs_ndas_2021050500.log get_obs_mrms_2021050500.log var_defns.sh.txt FV3LAM_wflow.xml.txt