Fix Multivariate MODE to log it's output to a single log file.

JohnHalleyGotway commented 3 years ago

Describe the Enhancement

This task is closely related to MET #1283. If all Multivariate MODE functionality is run by a single process, then the logging issue effectively goes away.

MET #1184 was included in the met-10.0.0-beta2 release to add the multi-variate MODE option. This version of the code spawns new processes to execute individual calls to the standard MODE prior to combining the output into a super object. This task is to figure out how to handle the writing of a log file in this context. Since multiple processes are spawned, having them write to the same output log file is non-trivial.

Since the calls to MODE are done sequentially, consider adding a no-clobber option to the logger so that each call to single MODE can open and append to the same log file without overwriting it.

Time Estimate

Estimate the amount of work required here. Issues should represent approximately 1 to 3 days of work.

Sub-Issues

Consider breaking the enhancement down into sub-issues. No sub-issues required.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

Define the source of funding and account keys here or state NONE.

Define the Metadata

Assignee

[ ] Select engineer(s) or no engineer required
[ ] Select scientist(s) or no scientist required

Labels

[x] Select component(s)
[x] Select priority
[x] Select requestor(s)

Projects and Milestone

[x] Review projects and select relevant Repository and Organization ones or add "alert:NEED PROJECT ASSIGNMENT" label
[x] Select milestone to next major version milestone or "Future Versions"

Define Related Issue(s)

Consider the impact to the other METplus components.

[x] METplus, MET, METdatadb, METviewer, METexpress, METcalcpy, METplotpy No impacts.

Enhancement Checklist

See the METplus Workflow for details.

[ ] Complete the issue definition above, including the Time Estimate and Funding Source.
[ ] Fork this repository or create a branch of develop. Branch name: feature_<Issue Number>_<Description>
[ ] Complete the development and test your changes.
[ ] Add/update log messages for easier debugging.
[ ] Add/update unit tests.
[ ] Add/update documentation.
[ ] Push local changes to GitHub.
[ ] Submit a pull request to merge into develop. Pull request: feature <Issue Number> <Description>
[ ] Define the pull request metadata, as permissions allow. Select: Reviewer(s), Project(s), Milestone, and Linked issues
[ ] Iterate until the reviewer(s) accept and merge your changes.
[ ] Delete your fork or branch.
[ ] Close this issue.

JohnHalleyGotway commented 2 years ago

Please note that any changes to how Multi-Variate MODE should be run based on this issue should be noted on the METplus issue for the Multi-Variate MODE wrapper (dtcenter/METplus#1585).

davidalbo commented 9 months ago

Closing this issue as it was fixed by refactoring multivariate mode to not spawn multiple instance of traditional mode. The newer design does not do this, it does sequential steps that depend on each other (so the original design no longer made sense).

dtcenter / MET