Thoughts on aligning repository architecture

[trexfeathers]

This repository is our first effort to re-use repository architecture so we don't have to repeat things across multiple repositories. However there are several other elements outside GHA workflows that would also benefit from sharing/alignment:

• Pre-commit
• Nox/Tox
• Benchmarking

Unlike GHA there is no established architecture for sharing this stuff across repositories, and it would be enormous scope creep to try and develop such a thing ourselves.

An alternative could be to maintain a Template repository, possibly this one if it had a different name, or perhaps a dedicated repository would be better as it could demonstrate how to reference SciTools/workflows. A monthly GHA could create pull requests comparing each of our repositories to the Template, giving a timely reminder for devs to do two things:

• Update the Template repository with any recent good ideas introduced elsewhere.
• Update each repository to align with the Template.

The automated pull requests themselves would not be be merged as-is, as there will be some legitimate differences such as repository names, or certain configs (e.g. version_scheme = "release-branch-semver" for setuptools_scm).

This idea isn't fully formed but it's so far the best one I have come up with for helping with this age-old problem.

[pp-mo]

there are several other elements outside GHA workflows [[e.g. pre-commit + tox]], ... Unlike GHA there is no established architecture for sharing this stuff across repositories,

I guess we could wrap up specific usages in GHA workflows, but the problem is how to inherit common configuration settings?

Tools controls are specified either in function-specific config files, or sometimes as sections in more general-purpose files. Many tools require one set of config for the tool itself, and another place where it is "hooked" into an invocation mechanism. At present, noxfile.py gathers some 'actions' code in one place, and pyproject.toml gathers some multiple tools configs.

E.G. for iris currently, tools config and scripting logic which we might conceivably wish to share across multiple projects currently occurs in ...

• .readthedocs.yml
• noxfile.py : tests; RTD including doctests + linkcheck; wheel install; benchmarks
• Makefile : lock files update
• .pre-commit-config.yaml : github PR checks; black and isort
• .gitignore
• .gitattributes and .git_archival.txt : for setuptools scm
• setup.cfg (and setup.py) : setuptools and flake8
• pyproject.toml : config sections for setuptools-scm, black, isort, pytest, coverage

I think this is all seriously bewildering! And every time we try to simplify things, the problem seems to grow back as we adopt more repo-level automation features 😢

It's almost like we need to generate all this file content by some general automated method, not least because some files contain config content for more than one tool.

[pp-mo]

some general automated method

always one more level of indirection !

[bjlittle]

This repo shouldn't be a template, that wouldn't really make sense. It provides the common GHAs, and other client scitools repos use those GHAs.

However, it would make sense to have a separate template repo that would bootstrap new scitools repos with some sensible common infrastructure content.

I'd imagine basic customisation would be required by the client repo, but it would be an 80/20 win on effort. It may also involve removal of inherited template content, but IMHO it's easier to remove than create correct content, so that's still a win.

[trexfeathers]

See https://github.com/SciTools/.github/tree/main/templates