Bugfix: ASCII2NC file window bad default value and redundant initialization of wrappers

georgemccabe commented 4 months ago

While testing the METplus 6.0.0-beta3 release, @malloryprow discovered a bug:

One of my jobs failed and jere is the output I am seeing

03/21 13:20:25.845 metplus.bf196009 (config_metplus.py:622) DEBUG: Setting [config] ASCII2NC_FILE_WINDOW_BEGIN to default value (empty string)
03/21 13:20:25.845 metplus.bf196009 (config_metplus.py:622) DEBUG: Setting [config] OBS_FILE_WINDOW_BEGIN to default value (0)
03/21 13:20:25.846 metplus.bf196009 (config_metplus.py:622) DEBUG: Setting [config] ASCII2NC_FILE_WINDOW_END to default value (empty string)
03/21 13:20:25.846 metplus.bf196009 (config_metplus.py:622) DEBUG: Setting [config] OBS_FILE_WINDOW_END to default value (0)
03/21 13:20:25.847 metplus.bf196009 (config_validate.py:174) DEBUG: Checking for deprecated environment variables in: /apps/ops/para/libs/intel/19.1.3.304/metplus/6.0.0-beta3/parm/met_config/Ascii2NcConfig_wrapped
03/21 13:20:25.851 metplus.bf196009 (config_metplus.py:852) ERROR: [config] ASCII2NC_FILE_WINDOW_BEGIN does not match expected format. Valid options match 3600, 3600S, 60M, or 1H

03/21 13:20:25.851 metplus.bf196009 (config_metplus.py:852) ERROR: [config] ASCII2NC_FILE_WINDOW_END does not match expected format. Valid options match 3600, 3600S, 60M, or 1H

ASCII2NC_FILE_WINDOW_BEGIN/END is not set in the config file being used (ASCII2NC_obsNDBC.conf), so the default for those settings kicked in, which is an empty string. However, there was an ERROR that the format for these settings did not match the expected format. It seems a mismatch that a default value for a setting does not match the expected format.

Describe the Problem

A quick solution would be to change the logic in ascii2nc_wrapper that handles file_window variables to use the function that handles it from command_builder. However, it appears that the wrapper initialization is run twice -- once when checking for deprecated environment variables and again to set up the wrapper to run. The first initialization is run just to get the class variable that holds the list of deprecated environment variables. That logic should be updated to read the class variable without initializing the wrapper, since a class variable can be read without creating an instance of it.

Expected Behavior

Running an ASCII2NC use case with ASCII2NC_FILE_WINDOW_BEGIN and ASCII2NC_FILE_WINDOW_END unset should run without error.

Environment

Describe your runtime environment: 1. Machine: (e.g. HPC name, Linux Workstation, Mac Laptop) 2. OS: (e.g. RedHat Linux, MacOS) 3. Software version number(s)

To Reproduce

Describe the steps to reproduce the behavior: 1. Go to '...' 2. Click on '....' 3. Scroll down to '....' 4. See error Post relevant sample data following these instructions: https://dtcenter.org/community-code/model-evaluation-tools-met/met-help-desk#ftp

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

Define the source of funding and account keys here or state NONE.

Define the Metadata

Assignee

[ ] Select engineer(s) or no engineer required
[ ] Select scientist(s) or no scientist required

Labels

[ ] Select component(s)
[ ] Select priority
[ ] Select requestor(s)

Projects and Milestone

[ ] Select Organization level Project for support of the current coordinated release
[ ] Select Repository level Project for development toward the next official release or add alert: NEED CYCLE ASSIGNMENT label
[ ] Select Milestone as the next bugfix version

Define Related Issue(s)

Consider the impact to the other METplus components.

[ ] METplus, MET, METdataio, METviewer, METexpress, METcalcpy, METplotpy

Bugfix Checklist

See the METplus Workflow for details.

[ ] Complete the issue definition above, including the Time Estimate and Funding Source.
[ ] Fork this repository or create a branch of main_\. Branch name: bugfix_<Issue Number>_main_<Version>_<Description>
[ ] Fix the bug and test your changes.
[ ] Add/update log messages for easier debugging.
[ ] Add/update unit tests.
[ ] Add/update documentation.
[ ] Add any new Python packages to the METplus Components Python Requirements table.
[ ] Push local changes to GitHub.
[ ] Submit a pull request to merge into main_\. Pull request: bugfix <Issue Number> main_<Version> <Description>
[ ] Define the pull request metadata, as permissions allow. Select: Reviewer(s) and Development issues Select: Organization level software support Project for the current coordinated release Select: Milestone as the next bugfix version
[ ] Iterate until the reviewer(s) accept and merge your changes.
[ ] Delete your fork or branch.
[ ] Complete the steps above to fix the bug on the develop branch. Branch name: bugfix_<Issue Number>_develop_<Description> Pull request: bugfix <Issue Number> develop <Description> Select: Reviewer(s) and Development issues Select: Repository level development cycle Project for the next official release Select: Milestone as the next official version
[ ] Close this issue.

PerryShafran-NOAA commented 4 months ago

@georgemccabe Also please note that the following was also written out as part of this bug (or separate) - but also saw this using the beta versions:

PB2NC_SKIP_TIMES is deprecated. Please use PB2NC_SKIP_VALID_TIMES

Is this something we should be changing on our end?

Thanks!

Perry

georgemccabe commented 4 months ago

@PerryShafran-NOAA, yes, you should change any instance of PB2NC_SKIP_TIMES to PB2NC_SKIP_VALID_TIMES to resolve those warnings. The wrappers now support skipping times based in initialization times as well, so the config variables were renamed to be more explicit.

PerryShafran-NOAA commented 4 months ago

@georgemccabe Thank you! Here are some other warnings noticed during our test:

WARNING: Both POINT_STAT_CLIMO_MEAN_INPUT_TEMPLATE and POINT_STAT_CLIMO_MEAN_FILE_NAME are set. Using value set in POINT_STAT_CLIMO_MEAN_FILENAME (/lfs/h2/emc/vpppg/noscrub/emc.vpppg/EVS/fix/climos/atmos/era5/mean{valid?fmt=%m%d})

WARNING: Both POINT_STAT_CLIMO_STDEV_INPUT_TEMPLATE and POINT_STAT_CLIMO_STDEV_FILE_NAME are set. Using value set in POINT_STAT_CLIMO_STDEV_FILENAME (/lfs/h2/ emc/vpppg/noscrub/emc.vpppg/EVS/fix/climos/atmos/era5/stdev{valid?fmt=%m%d}).
WARNING: Both GRID_STAT_CLIMO_MEAN_INPUT_TEMPLATE and GRID_STAT_CLIMO_MEAN_FILE_NAME are set. Using value set in GRID_STAT_CLIMO_MEAN_FILE_NAME (/lfs/h2/emc/vpppg/noscrub/emc.vpppg/EVS/fix/climos/atmos/ccpa/ccpa_24hr_1p0prob{valid?fmt=%m%d}.grib2).

Can you comment on these?

malloryprow commented 4 months ago

Should we open a GitHub Discussion for these? These aren't related to the ASCII2NC file window issue.

georgemccabe commented 4 months ago

Yes, @PerryShafran-NOAA , could you please create a new METplus Discussion and I can address these warnings there?

PerryShafran-NOAA commented 4 months ago

Done. These have already been commented on in a separate email exchange, but the discussion is better had on the METplus discussion page.

georgemccabe commented 4 months ago

The warnings relating to the climo variables ending with _INPUT_TEMPLATE and _FILE_NAME can appear due to the bug described in this issue when the _FILE_NAME variables are not set.

dtcenter / METplus