Make TC Analysis parallel

[forsyth2]

Issue resolution

• May close #180. If the errors from #180 persist, I will try merging the zppy implementation of https://github.com/E3SM-Project/e3sm_diags/pull/824 first, and run the tests again, seeing if they are gone then.
• Resolves the e3sm_diags piece of #623, thus closing #622.

Select one: This pull request is...

• [ ] a bug fix: increment the patch version
• [x] a small improvement: increment the minor version
• [ ] an incompatible (non-backwards compatible) API change: increment the major version

1. Does this do what we want it to do?

Objectives:

• Remove sequential-only restriction on tc_analysis task.
• Remove sequential-only restriction on the e3sm_diags task when tc_analysis is in sets.
• Change min-case tests of tc_analysis to use v2, since data in v3 is unavailable at the moment (https://github.com/E3SM-Project/e3sm_diags/pull/866#issuecomment-2405873119)

Required:

• [x] Product Management: I have confirmed with the stakeholders that the objectives above are correct and complete.
  • Discussed these design decisions with @chengzhuzhang
• [x] Testing: I have added or modified at least one "min-case" configuration file to test this change. Every objective above is represented in at least one cfg.
• [x] Testing: I have considered likely and/or severe edge cases and have included them in testing.

2. Are the implementation details accurate & efficient?

Required:

• [x] Logic: I have visually inspected the entire pull request myself.
• [ ] Logic: I have left GitHub comments highlighting important pieces of code logic. I have had these code blocks reviewed by at least one other team member.
  • Left comments, no block here requires review.

3. Is this well documented?

Required:

• [ ] Documentation: by looking at the docs, a new user could easily understand the functionality introduced by this pull request.
  • The vast majority of zppy's user documentation is via https://docs.e3sm.org/zppy/_build/html/main/parameters.html, which directs to https://github.com/E3SM-Project/zppy/blob/main/zppy/templates/default.ini, which doesn't appear to make any references to only being able to run tc_analysis sequentially.

4. Is this code clean?

Required:

• [x] Readability: The code is as simple as possible and well-commented, such that a new team member could understand what's happening.
• [x] Pre-commit checks: All the pre-commits checks have passed.

If applicable:

• [x] Software architecture: I have discussed relevant trade-offs in design decisions with at least one other team member. It is unlikely that this pull request will increase tech debt.
  • Discussed these design decisions with @chengzhuzhang

[forsyth2]

Unit tests pass (https://github.com/E3SM-Project/zppy/actions/runs/11355423070/job/31584722994?pr=631).

I didn't run into any of the issues from #180. Those errors were intermittent, so I do have some concern they just didn't happen to happen this time. That said, it's very possible underlying things have changed since those bugs were first noticed. For reference, those bugs were:

• GenerateConnectivityFile: error while loading shared libraries: libnetcdf.so.11: cannot open shared object file: No such file or directory previously occurred when multiple year sets of tc_analysis were run simultaneously. This was tested here with test_min_case_e3sm_diags_tc_analysis_parallel_chrysalis.cfg.
• socket.gaierror: [Errno -2] Name or service not known previously occurred in E3SM Diags when running the tc_analysis set in parallel. This was also tested here with test_min_case_e3sm_diags_tc_analysis_parallel_chrysalis.cfg.

Full testing details

``` # zppy dev environment mamba clean --all mamba env create -f conda/dev.yml -n zppy_dev_make-tc-parallel-20241015 conda activate zppy_dev_make-tc-parallel-20241015 pre-commit run --all-files pip install . python -u -m unittest tests/test_*.py # e3sm_diags dev environment cd ../e3sm_diags git checkout main git fetch upstream git reset --hard upstream/main git log # Should match https://github.com/E3SM-Project/e3sm_diags/commits/main mamba clean --all mamba env create -f conda-env/dev.yml -n e3sm_diags_20241015 conda activate e3sm_diags_20241015 pip install . cd ../zppy conda activate zppy_dev_make-tc-parallel-20241015 pip install . zppy -c tests/integration/generated/test_min_case_tc_analysis_simultaneous_1_chrysalis.cfg zppy -c tests/integration/generated/test_min_case_tc_analysis_simultaneous_2_chrysalis.cfg cd /lcrc/group/e3sm/ac.forsyth2/zppy_min_case_tc_analysis_simultaneous_1_output/test-631-v2/v2.LR.historical_0201/post/scripts/ grep -v "OK" *status # No errors cd /lcrc/group/e3sm/ac.forsyth2/zppy_min_case_tc_analysis_simultaneous_2_output/test-631-v2/v2.LR.historical_0201/post/scripts/ grep -v "OK" *status # No errors zppy -c tests/integration/generated/test_min_case_e3sm_diags_tc_analysis_chrysalis.cfg cd /lcrc/group/e3sm/ac.forsyth2/zppy_min_case_e3sm_diags_tc_analysis_output/test-631-v2/v2.LR.historical_0201/post/scripts grep -v "OK" *status # No errors # https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/zppy_min_case_e3sm_diags_tc_analysis_www/test-631-v2/v2.LR.historical_0201/e3sm_diags/atm_monthly_180x360_aave_tc_analysis/model_vs_obs_1985-1986/viewer/tc_analysis/index.html has plots zppy -c tests/integration/generated/test_min_case_e3sm_diags_tc_analysis_mvm_1_chrysalis.cfg # Wait to complete cd /lcrc/group/e3sm/ac.forsyth2/zppy_min_case_e3sm_diags_tc_analysis_mvm_1_output/test-631-v2/v2.LR.historical_0201/post/scripts/ grep -v "OK" *status # No errors cd - zppy -c tests/integration/generated/test_min_case_e3sm_diags_tc_analysis_mvm_2_chrysalis.cfg cd /lcrc/group/e3sm/ac.forsyth2/zppy_min_case_e3sm_diags_tc_analysis_mvm_2_output/test-631-v2/v2.LR.historical_0201/post/scripts/ grep -v "OK" *status # No errors # https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/zppy_min_case_e3sm_diags_tc_analysis_mvm_2_www/test-631-v2/v2.LR.historical_0201/e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_1995-1996/viewer/tc_analysis/index.html has plots zppy -c tests/integration/generated/test_min_case_e3sm_diags_tc_analysis_parallel_chrysalis.cfg cd /lcrc/group/e3sm/ac.forsyth2/zppy_min_case_e3sm_diags_tc_analysis_parallel_output/test-631-v2/v2.LR.historical_0201/post/scripts/ grep -v "OK" *status # No errors # https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/zppy_min_case_e3sm_diags_tc_analysis_parallel_www/test-631-v2/v2.LR.historical_0201/e3sm_diags/atm_monthly_180x360_aave_tc_analysis/model_vs_obs_1985-1986/viewer/tc_analysis/index.html has plots # https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/zppy_min_case_e3sm_diags_tc_analysis_parallel_www/test-631-v2/v2.LR.historical_0201/e3sm_diags/atm_monthly_180x360_aave_tc_analysis/model_vs_obs_1987-1988/viewer/tc_analysis/index.html has plots ```

[chengzhuzhang]

@forsyth2 thank you for working on the PR and the comments. I visually checked and it looks good. I'm happy to review any upcoming TC analysis related PRs, so feel free to tag me.

[forsyth2]

Thanks @chengzhuzhang!