Stage 1 Develop minimum `um2nc` for esm1.5 postprocessing

[truth-quark]

This is a high level project management issue to outline general work to prepare a minimum um2nc utility for esm1.5 post-processing. For simplicity, this content is a GitHub issue, however can be a separate markdown document if required.

The stages are intended as scaffolding, not a prescriptive approach (e.g. stage 2 & stage 3 may interleave).

Overview

Multiple versions of um2nc exist on gadi, with uncertain provenance. This project is to create a single canonical version of um2nc to replace other copies, starting from Martin Dix's version copied from 16/6/2024.

Background: the original um2nc code lacked unit tests & it is unclear what manual testing had been carried out (or the results). There was no way to determine if the code was functioning as intended. Additionally, no requirements documentation existed at the beginning of this project. Martin Dix knows project history & requirements, although this was not captured in documentation.

An early review identified non-modular code as another primary concern. The code had been developed over several years, accumulating features across the module. The base version contained a mix of smaller, separate functions & several larger workflow functions. The larger workflow functions bundled several functional steps together, with the coupling making the code difficult or unwieldy to test isolated parts. The code lacked a consistent software architecture. Other smaller issues existed, including complicated logic, older features & missing checks.

Naming: different copies have different module names, depending on different authors & their customisations. In this repo, the name um2nc is considered synonymous with um2netcdf, with the primary module being um2netcdf.py.

Overarching Requirements & Goals

• Deduplicate um2nc copies with this canonical version
• Code should be tested before inclusion in releases
  • Undertake Test Driven Repair (TDR) to retrofit unit testing
  • Improve code design, structure & modularity
  • Ensure script(s) can execute
• Create a reference best practices project to demonstrate high quality software
• Provide an example of unit testing in practice

Development Stage 1 - Retrofitting & Modularisation

The first dev stage focuses on rearranging the code & retrofitting unit tests. The goal is to substantially modularise the code base & provide a suite of tests to support future fixes & refactoring. Without tests, there is no safety net to determine if the changes are breaking or non-breaking.

NB: as this stage is uncertain, some loose work is perfectly acceptable. The goal is to decouple first, learn the code & then improve it. It is significantly easier working with smaller pieces due to reduced dependencies.

Tasks:

• [x] Document functional requirements in issues
• [ ] Functional decomposition for modularity
  • [x] Extract responsibilities to small functions
  • [ ] Retrofit unit tests to the extracted functions (& to smaller existing functions)
  • [ ] Rework high level functions to manage workflows through delegation
• [ ] Keep code functionally similar while restructuring for modularity
  • Some code neatening is fine
  • Initially avoid large changes/breaking binary equivalence until tests are in place
  • Fixes can proceed when managed via stage 2/3
  • Reordering code/workflows may be necessary to allow testing
  • Defer substantial changes/repairs/refactoring unless necessary
• [ ] Raise test coverage to a decent level
  • Cover the most common code paths
  • Avoid 100% coverage, some gaps are acceptable, necessary & prudent
  • Identify gaps / defer if requirements unclear (fix when problems are better understood)
  • Identify bugs
  • Identify new features/enhancements & add as issues
• [ ] Add basic integration test script(s) to run um2nc with real data
  • This supports unit testing
• [ ] Base CI setup

[truth-quark]

This requirement also relates to this issue: https://github.com/ACCESS-NRI/access-esm1.5-configs/issues/3

[truth-quark]

See also #65 / the Test Driven Repair strategy for further details.

[truth-quark]

This issue is being split into separate issues for the 3 core development steps.