Replace italics below with details for this issue.
Describe the Problem
A METplus error occurs when this EVS/cam job is run on NCEP's WCOSS2 system in one environment (named "test_ecflow_env" for testing here). However, the same job is successful when run in a different environment ("test_parallel_env"). In the first case, METplus is unable to find a file that exists when checked manually. Could you help us debug?
The above excerpt is a section of our code that checks for the presence of an observation file that will be used in METplus, a check which was successful in this case. To confirm, a list command for the file showed that the file does exist on the system (WCOSS2/Dogwood).
However, the METplus run that followed the excerpted code outputs an error. The associated METplus log is attached, but here is an excerpt:
32 11/08 18:36:50.387 metplus (command_builder.py:694) DEBUG: Looking for OBS_INPUT files under /lfs/h2/emc/vpppg/noscrub/emc.vpppg/com/evs/v1.0/prep/cam within range [-300,300] using template mrms.{valid?fmt=%Y%m%d}/conus/EchoTop18_00.50_{valid?fmt=%Y%m%d}-{valid?fmt=%H}0000.G227.nc
33 11/08 18:36:50.792 metplus (command_builder.py:257) ERROR: (command_builder.py:741) Could not find OBS_INPUT files under /lfs/h2/emc/vpppg/noscrub/emc.vpppg/com/evs/v1.0/prep/cam within range [-300,300] using template mrms.{valid?fmt=%Y%m%d}/conus/EchoTop18_00.50_{valid?fmt=%Y%m%d}-{valid?fmt=%H}0000.G227.nc
34 11/08 18:36:50.793 metplus (command_builder.py:257) ERROR: (command_builder.py:490) Could not find observation file
The valid time in this case is 2023110715, so METplus should have been able to find the file.
We also confirmed that (1) the observation file was already available before METplus was run, at about 11/08 18:15Z and (2) the valid time stored in the observation file is 11/07/2023 15:00:38Z, which should have satisfied the matching condition in METplus.
I've set up two test cases ("test_ecflow_env" and "test_parallel_env") with slightly different environments, one in which the error described here can be replicated, the other in which the same METplus run is successful.
I'm wondering if any part of our configuration jumps out to you as a possible cause of the issue?
Expected Behavior
The observation file should satisfy the METplus check for files that match the template within the valid hour range (11/07/2023 15Z +/- 5 mins). That observation file should be used to complete the METplus run successfully (the forecast file having been already found successfully). In other words, the "test_parallel_env" test results represent the expected behavior.
Environment
Describe your runtime environment:
Machine: WCOSS2, Dogwood
OS: Linux
Here are the modules loaded into the environment from which METplus was run in both test cases:
Describe the steps to reproduce the behavior:
Two very similar tests may help isolate the problem. The first fails and the second succeeds. To test:
1) Log onto WCOSS2 Dogwood (prod) and cd into your work space (a test directory on ptmp or stmp will work)
2) Transfer the following files (attached here) from your local computer to your work space on WCOSS2 (Dogwood):
test_ecflow_env.txt
test_parallel_env.txt
3) Convert the files to bash scripts:
cp test_ecflow_env.txt test_ecflow_env.sh
cp test_parallel_env.txt test_parallel_env.sh
4) Make sure the files are user-executable:
chmod u+x test_ecflow_env.sh
chmod u+x test_parallelenv.sh
5) Open each test file (vi test*_env.sh) and edit the section below #### EDIT BELOW FOR TESTING #### as needed
6) Run each test script, redirecting the output
test_ecflow_env.sh &> test1.out
test_parallel_env.sh &> test2.out
(Note: these will write two data directories for METplus to use
7) test_ecflow_env should output errors, failing to find the output file. test_parallel_env should succeed.
If testing this on WCOSS2/Dogwood (currently the production machine) is not possible, or if you'd prefer the test data is provided here, please let me know and I can ask someone to copy it over. Thanks
Relevant Deadlines
The relevant EVS code will be delivered at COB 11/17/2023.
Funding Source
NOAA/NCEP/EMC/VPPPG
Define the Metadata
Assignee
[ ] Select engineer(s) or no engineer required
[ ] Select scientist(s) or no scientist required
Labels
[ ] Select component(s)
[ ] Select priority
[ ] Select requestor(s)
Projects and Milestone
[ ] Select Organization level Project for support of the current coordinated release
[ ] Select Repository level Project for development toward the next official release or add alert: NEED CYCLE ASSIGNMENT label
[ ] Select Milestone as the next bugfix version
Define Related Issue(s)
Consider the impact to the other METplus components.
[ ] Submit a pull request to merge into main_\.
Pull request: bugfix <Issue Number> main_<Version> <Description>
[ ] Define the pull request metadata, as permissions allow.
Select: Reviewer(s) and Development issues
Select: Organization level software support Project for the current coordinated release
Select: Milestone as the next bugfix version
[ ] Iterate until the reviewer(s) accept and merge your changes.
[ ] Delete your fork or branch.
[ ] Complete the steps above to fix the bug on the develop branch.
Branch name: bugfix_<Issue Number>_develop_<Description>
Pull request: bugfix <Issue Number> develop <Description>
Select: Reviewer(s) and Development issues
Select: Repository level development cycle Project for the next official release
Select: Milestone as the next official version
Replace italics below with details for this issue.
Describe the Problem
A METplus error occurs when this EVS/cam job is run on NCEP's WCOSS2 system in one environment (named "test_ecflow_env" for testing here). However, the same job is successful when run in a different environment ("test_parallel_env"). In the first case, METplus is unable to find a file that exists when checked manually. Could you help us debug?
Attached are copies of (1) the stdout log for the EVS job that failed (2) the METplus log file of the failed METplus run, and (3) two test scripts, described below and in the "To Reproduce" section. EVS_fail_job.txt METplus_fail_log.txt test_ecflow_env.txt test_parallel_env.txt
Here is an excerpt from the stdout log for the EVS job that included that METplus run:
The above excerpt is a section of our code that checks for the presence of an observation file that will be used in METplus, a check which was successful in this case. To confirm, a list command for the file showed that the file does exist on the system (WCOSS2/Dogwood).
However, the METplus run that followed the excerpted code outputs an error. The associated METplus log is attached, but here is an excerpt:
The valid time in this case is 2023110715, so METplus should have been able to find the file.
We also confirmed that (1) the observation file was already available before METplus was run, at about 11/08 18:15Z and (2) the valid time stored in the observation file is 11/07/2023 15:00:38Z, which should have satisfied the matching condition in METplus.
I've set up two test cases ("test_ecflow_env" and "test_parallel_env") with slightly different environments, one in which the error described here can be replicated, the other in which the same METplus run is successful.
I'm wondering if any part of our configuration jumps out to you as a possible cause of the issue?
Expected Behavior
The observation file should satisfy the METplus check for files that match the template within the valid hour range (11/07/2023 15Z +/- 5 mins). That observation file should be used to complete the METplus run successfully (the forecast file having been already found successfully). In other words, the "test_parallel_env" test results represent the expected behavior.
Environment
Describe your runtime environment:
To Reproduce
Describe the steps to reproduce the behavior: Two very similar tests may help isolate the problem. The first fails and the second succeeds. To test: 1) Log onto WCOSS2 Dogwood (prod) and cd into your work space (a test directory on ptmp or stmp will work) 2) Transfer the following files (attached here) from your local computer to your work space on WCOSS2 (Dogwood): test_ecflow_env.txt test_parallel_env.txt 3) Convert the files to bash scripts: cp test_ecflow_env.txt test_ecflow_env.sh cp test_parallel_env.txt test_parallel_env.sh 4) Make sure the files are user-executable: chmod u+x test_ecflow_env.sh chmod u+x test_parallelenv.sh 5) Open each test file (vi test*_env.sh) and edit the section below #### EDIT BELOW FOR TESTING #### as needed 6) Run each test script, redirecting the output test_ecflow_env.sh &> test1.out test_parallel_env.sh &> test2.out (Note: these will write two data directories for METplus to use 7) test_ecflow_env should output errors, failing to find the output file. test_parallel_env should succeed.
If testing this on WCOSS2/Dogwood (currently the production machine) is not possible, or if you'd prefer the test data is provided here, please let me know and I can ask someone to copy it over. Thanks
Relevant Deadlines
The relevant EVS code will be delivered at COB 11/17/2023.
Funding Source
NOAA/NCEP/EMC/VPPPG
Define the Metadata
Assignee
Labels
Projects and Milestone
Define Related Issue(s)
Consider the impact to the other METplus components.
Bugfix Checklist
See the METplus Workflow for details.
bugfix_<Issue Number>_main_<Version>_<Description>
bugfix <Issue Number> main_<Version> <Description>
bugfix_<Issue Number>_develop_<Description>
Pull request:bugfix <Issue Number> develop <Description>
Select: Reviewer(s) and Development issues Select: Repository level development cycle Project for the next official release Select: Milestone as the next official version