Improve regression testing logic

[georgemccabe]

Notes from @JohnHalleyGotway from PR review #810:

• When differences are detected, add the files from the reference "truth" branch to the artifact created for that pull request GHA. That'll make it really easy for the PR reviewer to investigate the diffs.

• Can we make the docker data volumes more self-describing? If possible, add a "/README" file that lists info like the creation date, source branch or SHA. There's potential to easily lose track of what data volumes where created by what version of the code and when.

• Recommend mechanism to list files that should be EXCLUDED from the diffing logic. Could be as simple as a text file listing files to be skipped. I'm sure this will come up and be useful.

Describe the Enhancement

See above

Time Estimate

Estimate the amount of work required here. Issues should represent approximately 1 to 3 days of work.

Sub-Issues

Consider breaking the enhancement down into sub-issues.

• [ ] Add a checkbox for each sub-issue here.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

Define the source of funding and account keys here or state NONE.

Define the Metadata

Assignee

• [ ] Select engineer(s) or no engineer required
• [ ] Select scientist(s) or no scientist required

Labels

• [X] Select component(s)
• [X] Select priority
• [X] Select requestor(s)

Projects and Milestone

• [X] Review projects and select relevant Repository and Organization ones or add "alert:NEED PROJECT ASSIGNMENT" label
• [X] Select milestone to next major version milestone or "Future Versions"

Define Related Issue(s)

Consider the impact to the other METplus components.

• [X] METplus, MET, METdatadb, METviewer, METexpress, METcalcpy, METplotpy

Enhancement Checklist

See the METplus Workflow for details.

• [ ] Complete the issue definition above, including the Time Estimate and Funding Source.
• [ ] Fork this repository or create a branch of develop. Branch name: feature_<Issue Number>_<Description>
• [ ] Complete the development and test your changes.
• [ ] Add/update log messages for easier debugging.
• [ ] Add/update unit tests.
• [ ] Add/update documentation.
• [ ] Push local changes to GitHub.
• [ ] Submit a pull request to merge into develop. Pull request: feature <Issue Number> <Description>
• [ ] Define the pull request metadata, as permissions allow. Select: Reviewer(s), Project(s), Milestone, and Linked issues
• [ ] Iterate until the reviewer(s) accept and merge your changes.
• [ ] Delete your fork or branch.
• [ ] Close this issue.

[georgemccabe]

Update: When differences are found, the new output file, old truth file, and if applicable a difference file are added to an artifact with a name that starts with "diff" that can be downloaded to investigate closer.

[georgemccabe]

Improvements to the process to create Conda environments to use in the use case tests, as suggested by Bill Little from the UK MetOffice:

• Generally, it’s better to create an environment by asking conda to resolve and install all the packages at once, rather than doing it incrementally each package at a time.
• You could use a conda environment YAML file in the Docker context containing all the packages (with channels and package pinned versions) that you want to install at once.
• Use mamba instead of conda (much faster)
• Use conda-lock to ensure sub-dependencies of your specified top level dependencies do not change