Internal: Improve NetCDF file diff to match MET diff functionality

[georgemccabe]

This came up in the comments of issue #2704.

Describe the Enhancement

Improve the logic in the METplus diff code to catch the differences flagged by the MET diff tests but not in METplus. @JohnHalleyGotway will comment to provide some sample files to use for testing.

Time Estimate

Estimate the amount of work required here. Issues should represent approximately 1 to 3 days of work.

Sub-Issues

Consider breaking the enhancement down into sub-issues.

• [ ] Add a checkbox for each sub-issue here.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

@natalieb-noaa is funded through the DTC to work on METplus tasks.

Define the Metadata

Assignee

• [ ] Select engineer(s) or no engineer required
• [ ] Select scientist(s) or no scientist required

Labels

• [ ] Review default alert labels
• [ ] Select component(s)
• [ ] Select priority
• [ ] Select requestor(s)

Milestone and Projects

• [ ] Select Milestone as a METplus-Wrappers-X.Y.Z version, Consider for Next Release, or Backlog of Development Ideas
• [ ] For a METplus-Wrappers-X.Y.Z version, select the METplus-Wrappers-X.Y.Z Development project

Define Related Issue(s)

Consider the impact to the other METplus components.

• [ ] METplus, MET, METdataio, METviewer, METexpress, METcalcpy, METplotpy

Enhancement Checklist

See the METplus Workflow for details.

• [ ] Complete the issue definition above, including the Time Estimate and Funding Source.
• [ ] Fork this repository or create a branch of develop. Branch name: feature_<Issue Number>_<Description>
• [ ] Complete the development and test your changes.
• [ ] Add/update log messages for easier debugging.
• [ ] Add/update unit tests.
• [ ] Add/update documentation.
• [ ] Add any new Python packages to the METplus Components Python Requirements table.
• [ ] For any new datasets, an entry to the METplus Verification Datasets Guide.
• [ ] Push local changes to GitHub.
• [ ] Submit a pull request to merge into develop. Pull request: feature <Issue Number> <Description>
• [ ] Define the pull request metadata, as permissions allow. Select: Reviewer(s) and Development issue Select: Milestone as the next official version Select: METplus-Wrappers-X.Y.Z Development project for development toward the next official release
• [ ] Iterate until the reviewer(s) accept and merge your changes.
• [ ] Delete your fork or branch.
• [ ] Close this issue.

[JohnHalleyGotway]

Please note that differences to NetCDF MTD output file were flagged in this MET GHA Testing Workflow run. The diff.zip artifact can be found here: diff.zip

In that zip file compare: mtd/mtd_BASIC_20100517_010000V_obj_TRUTH.nc vs mtd/mtd_BASIC_20100517_010000V_obj_OUTPUT.nc

Where the order of the dimensions and details of the global attributes, as seen in output from ncdump -h. While the order of the dimensions and variables really do not matter, we really should be checking for differences in the global and variable attributes. In MET the comp_nc.R script calls the compareNc utility which calls ncdump -h followed by diff to check for NetCDF header diffs. I'd recommend using a Python-only solution rather than adding a dependency on ncdump, if possible and convenient.

[JohnHalleyGotway]

Reassigned this issue @natalieb-noaa so that she can work on it in conjunction with dtcenter/MET#2718, which updates MET to use the differencing logic provided by METplus. Seems like it'd be good to do these at the same time.