search

E3SM-Project / e3sm_diags

E3SM Diagnostics package
https://e3sm-project.github.io/e3sm_diags
BSD 3-Clause "New" or "Revised" License
41 stars 32 forks source link

[Feature]: Adapt CDAT Migration regression testing scripts as integration tests to replace image checker #894

Open tomvothecoder opened 3 days ago

tomvothecoder commented 3 days ago

Is your feature request related to a problem?

The current integration test uses image checking to compare pixels. These tests often break due to very subtle differences that are introduced externally through different versions of packages, especially matplotlib.

Describe the solution you'd like

The CDAT migration regression testing script adapts the run__...py script to run the sets. We then manually update a regression testing notebook to compare the results against the latest main (right now CDAT-based).

Note: These are rough details that need to be ironed out. What we need to do is:

  1. Automate the testing script
  2. Adapt the comparison notebook to a Python script and automate
  3. Determine subset to compare against main
  4. Run results on latest main (after cdat-migration-fy24 is merged) and store results on Perlmutter/LCRC
  5. Integrate with pytest and GH Actions build workflow

Describe alternatives you've considered

No response

Additional context

No response

chengzhuzhang commented 3 days ago

Thank you for creating this issue. I think we can use v3 simulation data for the complete run. There is a bash file I created for the 2024 tutorial which include the required input data for all sets. https://github.com/E3SM-Project/e3sm_diags/blob/14612912b66fb71ac0e82504034ded2e8e2aba71/examples/tutorials2024/e3sm_diags_extended.bash