[Refactor]: CDAT Migration - Add test run scripts and results

[tomvothecoder]

Is your feature request related to a problem?

We need a standard structure and process to streamline regression testing. We will be using perlmutter as our test machine.

1. Create a shared directory that stores results of test scripts for main and branches (@tomvothecoder) -- DONE

• Root directory: /global/cfs/projectdirs/e3sm/e3sm_diags_cdat_test/
• Example branch results directory: /global/cfs/projectsdirs/e3sm/e3sm_diags_cdat_test/655-zonal-mean-2d/branch

2. Run run_v2_9_0_all_sets_E3SM_machines.py on main to generate baseline results for testing (@tomvothecoder) -- DONE

• Directory: /global/cfs/projectdirs/e3sm/e3sm_diags_cdat_test/main

• [x] Add template test script to cdat-migration-fy24 with configurable results_dir

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

[tomvothecoder]

If the second step isn't what we need:

3. (OPTIONAL) Set up test scripts for each set (@chengzhuzhang)

Steps:

• [ ] Open a PR with these scripts to merge to cdat-migration-fy24
  • Need to store intermediary files and metrics files for comparison
• [ ] Store scripts under auxiliary_tools/cdat_regression_testing/run_scripts
• [ ] Think about test script methodology:
  • Best to test all possible variables, maybe with 1 season, regridding is variable dependent
  • Or select a subset of variables to speed up testing process

Needed for:

• [ ] polar
• [ ] cosp_histogram
• [x] zonal_mean_xy
• [ ] zonal_mean_2d and zonal_mean_2d_stratosphere
• [ ] area_mean_time_series
• [ ] annual_cycle_zonal_mean
• [ ] meridional_mean_2d
• [ ] enso_diags
• [ ] qbo
• [ ] streamflow
• [ ] diurnal_cycle
• [ ] arm_diags
• [ ] tc_analysis
• [ ] aerosol_aeronet
• [ ] aerosol_budget

Run test scripts using main branch (TBD)

• Store the results in the shared directory from 1.
• NOTE: we need to address #759 to have results more closely aligned with Xarray code
• Follow https://github.com/E3SM-Project/e3sm_diags/wiki/CDAT-Migration-FY24-%E2%80%90-General-Guide

[tomvothecoder]

I just added the template run script to cdat-migration-fy24 (related commit). This issue is now complete.