This PR refactors the integration tests to improve maintainability by eliminating redundant code and making tests more readable. It also adds the use_cfg flag to run_diags(). This flag determines whether to use .cfg file(s) specified either by (1) -d/--diags (2) default diags. The goal is to make it easier to debug only a few diagnostic sets by setting use_cfg=False.
Changes
Refactor integration tests (tests/integration)
Delete test_dataset.py because it is a really old and incomplete test file for the legacy Dataset class based on CDAT
Delete test_all_sets.py and all_sets_modified.cfg because it tests for expected image counts which is redundant (test_all_sets_image_diffs.py does this already)
Replace subprocess calls with direct call to Python API for real-time test results and easier debugging
Move complete_run.py to /test/integration
Move test_run.py to tests/e3sm_diags since it is more of a unit test file
Refactor Run class (run.py)
Closes #735
Add use_cfg boolean argument to run_diags() and get_run_parameters()
Add self.cfg_path attribute, which is set if .cfg file(s) are used for diagnostic runs
Add is_cfg_file_arg_set() property to check parser for -d/--diags
Rename get_final_parameters() to get_run_parameters() and refactored to smaller methods
Update CI/CD build workflow (_buildworkflow.yml)
Split up testing step into: 1) run unit tests 2) download integration test data 3) run integration tests
Easier to keep track of runtime and results
Example script
import os
from e3sm_diags.parameter.core_parameter import CoreParameter
from e3sm_diags.run import runner
param = CoreParameter()
# Location of the data.
param.test_data_path = "/global/cfs/cdirs/e3sm/e3sm_diags/test_model_data_for_acme_diags/time-series/E3SM_v1"
param.reference_data_path = "/global/cfs/cdirs/e3sm/e3sm_diags/test_model_data_for_acme_diags/time-series/E3SM_v1"
# Set this parameter to True.
# By default, e3sm_diags expects the test data to be climo data.
param.test_timeseries_input = True
# Years to slice the test data, base this off the years in the filenames.
param.test_start_yr = "2011"
param.test_end_yr = "2013"
# Set this parameter to True.
# By default, e3sm_diags expects the ref data to be climo data.
param.ref_timeseries_input = True
# Years to slice the ref data, base this off the years in the filenames.
param.ref_start_yr = "1850"
param.ref_end_yr = "1852"
# When running with time-series data, you don't need to specify the name of the data.
# But you should, otherwise nothing is displayed when the test/ref name is needed.
param.short_test_name = "historical_H1"
param.short_ref_name = "historical_H1"
# This parameter modifies the software to accommodate model vs model runs.
# The default setting for run_type is 'model_vs_obs'.
param.run_type = "model_vs_model"
# Name of the folder where the results are stored.
# Change `prefix` to use your directory.
prefix = "/global/cfs/cdirs/e3sm/www/<your directory>/examples"
param.results_dir = os.path.join(prefix, "ex1_modTS_vs_modTS_3years")
# Below are more optional arguments.
# What plotsets to run the diags on.
# If not defined, then all available sets are used.
param.sets = ["lat_lon"]
# What seasons to run the diags on.
# If not defined, diags are run on ['ANN', 'DJF', 'MAM', 'JJA', 'SON'].
param.seasons = ["ANN"]
# Title of the difference plots.
param.diff_title = "Model (2011-2013) - Model (1850-1852)"
# For running with multiprocessing.
# param.multiprocessing = True
# param.num_workers = 32
runner.sets_to_run = ["lat_lon"]
runner.run_diags([param], use_cfg=False)
Checklist
[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] My changes generate no new warnings
[x] Any dependent changes have been merged and published in downstream modules
If applicable:
[x] New and existing unit tests pass with my changes (locally and CI/CD build)
[x] I have added tests that prove my fix is effective or that my feature works
[x] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] I have noted that this is a breaking change for a major release (fix or feature that would cause existing functionality to not work as expected)
Description
Overview
This PR refactors the integration tests to improve maintainability by eliminating redundant code and making tests more readable. It also adds the
use_cfg
flag torun_diags()
. This flag determines whether to use.cfg
file(s) specified either by (1)-d/--diags
(2) default diags. The goal is to make it easier to debug only a few diagnostic sets by settinguse_cfg=False
.Changes
test_dataset.py
because it is a really old and incomplete test file for the legacyDataset
class based on CDATtest_all_sets.py
andall_sets_modified.cfg
because it tests for expected image counts which is redundant (test_all_sets_image_diffs.py
does this already)subprocess
calls with direct call to Python API for real-time test results and easier debuggingcomplete_run.py
to/test/integration
test_run.py
totests/e3sm_diags
since it is more of a unit test fileRun
class (run.py)use_cfg
boolean argument torun_diags()
andget_run_parameters()
self.cfg_path
attribute, which is set if.cfg
file(s) are used for diagnostic runsis_cfg_file_arg_set()
property to check parser for-d/--diags
get_final_parameters()
toget_run_parameters()
and refactored to smaller methodsExample script
Checklist
If applicable: