Write any intermediary files to a user-specified directory

[jprestop]

Describe the Task

In working on Issue #143 Ensure METplotpy installs on NOAA machines, it was discovered that the METplotpy code is attempting to write at least one intermediary file in the same location where the test data exists. For development at EMC, a common installation that is available for everyone to use is necessary. Eventually, NCO will install a version that would be available for operational use. The users will need to use yaml configuration files that have been copied into their own directories (or a single directory for the planned EMC verification system). So they will be able to handle changing paths to data files for their various model comparisons. Specifying the full path to the data files for the default plot configurations would be nice so that the default/canned test cases will work for any user. Ideally, there would be a configurable way of doing this so that it is not necessarily to manually change filenames every time METplotpy is installed.

The METplotpy code needs to be changed to write any intermediary files to either 1) the users' current working directory or 2) a user-specified directory. Writing to a user-specified directory gives the most flexibility and is the recommended way to proceed.

Time Estimate

Estimate the amount of work required here. Issues should represent approximately 1 to 3 days of work. To be filled in by assigned enginneer

Sub-Issues

Consider breaking the task down into sub-issues.

Provide support in config files and .py module to turn on/off writing of the dump file used by METviewer, and add the optional config option points_path (and the associated code in the .py file). Apply the changes to the following plots:

• [x] bar
• [x] box
• [x] ens_ss
• [x] equivalence_testing_bounds
• [x] line
• [x] performance diagram
• [x] reliability diagram
• [x] roc diagram
• [x] wind rose *Note, the newly added histogram plots have this support (see Github issue #141 )

Documentation:

• [x] bar
• [x] box
• [x] ens_ss
• equivalence_testing_bounds <- no documentation exists (see #173)
• histogram 2d<- no tests exist so no place to put config files and sample data (see #173)
• histogram <- no documentation exists (see #173)
• [x] line
• [x] performance diagram
• [x] reliability diagram
• [x] roc diagram
• wind rose <- no documentation exists (see #173)

Relevant Deadlines

METplotpy-1.1.0-beta4

Funding Source

Split 2793541, 2788881, 2700041, 2700042

Define the Metadata

Assignee

• [x] Select engineer(s) or no engineer required
• [ ] Select scientist(s) or no scientist required

Labels

• [x] Select component(s)
• [x] Select priority
• [x] Select requestor(s)

Projects and Milestone

• [x] Select Repository and/or Organization level Project(s) or add alert: NEED PROJECT ASSIGNMENT label
• [x] Select Milestone as the next official version or Future Versions

Define Related Issue(s)

Consider the impact to the other METplus components.

• [x] METplus, MET, METdatadb, METviewer, METexpress, METcalcpy, METplotpy

Task Checklist

See the METplus Workflow for details.

• [ ] Complete the issue definition above, including the Time Estimate and Funding Source.
• [ ] Fork this repository or create a branch of develop. Branch name: feature_<Issue Number>_<Description>
• [ ] Complete the development and test your changes.
• [ ] Add/update log messages for easier debugging.
• [ ] Add/update unit tests.
• [ ] Add/update documentation.
• [ ] Push local changes to GitHub.
• [ ] Submit a pull request to merge into develop. Pull request: feature <Issue Number> <Description>
• [ ] Define the pull request metadata, as permissions allow. Select: Reviewer(s) and Linked issues Select: Repository level development cycle Project for the next official release Select: Milestone as the next official version
• [ ] Iterate until the reviewer(s) accept and merge your changes.
• [ ] Delete your fork or branch.
• [ ] Close this issue.

[bikegeek]

What is the intermediary file? Which plot?

[jprestop]

@bikegeek Any intermediary files that are written by METplotpy. For an example, see @LoganDawson-NOAA's post here: https://github.com/dtcenter/METplotpy/issues/143#issuecomment-918562654

[bikegeek]

This is going to impact METviewer, I will need to check with Tatiana to see how METviewer will search for the points file. I don't know if we'll be able to get this done by beta3 with two other high priority issues still in progress

[jprestop]

@bikegeek Thanks for the update. Adding @TaraJensen to keep her in the loop.

[TatianaBurek]

@jprestop , @bikegeek , @fisherhucar
I identified the reason and found the place in the source code where METplotpy writes points file into the data directory. Reason: METviewer legacy workflow that is hard to change Location in the source code: multiple, in each class that extends BasePlot and implements write_output_file() method. Example: https://github.com/dtcenter/METplotpy/blob/847eb28c07236ffe38d545a98ef8bc603e30da33/metplotpy/plots/box/box.py#L527

Proposed solutions:

1. This is more a work around: do not create and write points files. In order to do that set dump_points_1 to False

2. Since METviewer expects .points files in the data directory we can't change the current implementation but instead should add an optional parameter to the config file 'points_path' for example. If this parameter is missing, Plot classes would write point to the data directory (current workflow). If the parameter is present - the file will be put into that location. Should be a quick fix.