G Infrastructure: Homogeneous build, test and deploy metrics

[cmutel]

Motivation

Every time we need to make a change, e.g. add a tool, update Github Workflows system images, etc. we need to manually change every library. Manual changes can be inconsistent and are a pain, leading to procrastination. Larger changes are basically impossible to imagine.

Desired template attributes

We need a standard template and guide for libraries which has the following qualities:

• Uses Github workflows to test all three OSes with both the latest and next-to-last system images and Py 3.7-3.10.
• Test installation using both conda and pip
• Uses setup.cfg instead of setup.py. Include sections for dev (has e.g. pytest, pip-tools), stable (all requirements pinned), and normal installation (requirements have ranges of >=)
• Reports 1-click code coverage results
• Provides badges for CI results, code coverage (e.g. shields.io)
• Has a solution for hosted HTML documentation including napoleon
• Uses tooling to help improve code quality
• Single-source the library version (why these solutions so hacky!?)
• Automated builds and uploads to anaconda on tagged commits

Adapting the hypermodern template

A lot of these attributes are found in the hypermodern template, though @cmutel would argue against the following:

• Use of poetry: Better to stick with standard tools instead of more opinionated ones in components designed to be integrated into applications.
• Use of tox or nox: Not necessary, we do testing in CI, and setting up a venv for each test run is quite heavy for our components.
• Automated build notes. Could be considered in the future, we don't yet have the discipline to use this consistently, and it makes life harder for new contributors.

We should use the following (copied from their docs):

• Documentation with Sphinx, MyST, and Read the Docs using the furo theme
• Automated uploads to PyPI and TestPyPI

pre-commit hooks

Note that we assume the following are done as pre-commit hooks by the template (instead of by Github Workflows):

• Import sorting with isort
• Code formatting with Black and Prettier
• Black formatting in documentation snippets
• Static type-checking with mypy
• Runtime type-checking with Typeguard

cookiecutter instead of pyscaffold

pyscaffold ticks a lot of boxes, but has some controversial choices which are hard to remove, including the use of setuptools-scm and tox, has custom commands instead of pip and friends (not great for new users), and would need to adapted to include the Azure Workflows config in any case. @cmutel thinks that a custom cookiecutter template is probably the best choice. Ideally, someone would already have such a template, as the cookiecutter ecosystem is large, but no suitable template was found browsing cookiecutter-templates.

pip, not poetry or pipenv

Let's skip the drama, pip with pip-tools and pipreqs works just fine.

Useful links

• Python packaging authority
• PyPA: Using Github Workflows to publish packages
• pip-tools
• pipreqs
• cookiecutter conda template
• setuptools on setup.cfg
• Nice intro to pyproject and setup.cfg
• Workflow from @romainsacchi which does automatic package building and uploading

Tasks

• [ ] Make cookiecutter template that satisfies the required attributes
• [ ] Document use of this template, both for new library creation and for updating libraries
• [ ] Create example project from the example template
• [ ] Have regular checks to ensure that all Brightway libraries work correctly as dependent libraries evolve

[tngTUDOR]

Starting with PEP 621, the Python community selected pyproject.toml as a standard way of specifying project metadata https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html

[tngTUDOR]

Repository cookiecutter-brightwaylib is a starting point. It contains already a basic cookiecutter template, and issues related to each of the specification requirements described here.

[cmutel]

Here is a job description to find a freelancer to work on this:

Update Python packaging and CI scripts for an open source framework for life cycle assessment

Brightway (https://brightway.dev/) is an open source framework for life cycle assessment. Our packaging concept and CI infrastructure needs to be updated to use modern tools and make our lives easier. Specifically, the following changes need to be made:

• Switch from setup.py to pyproject.toml. No need to switch to anything like poetry or any other fancy build system.

(The reason we are hiring someone to do this in the first place is that we can build libraries using pyproject.toml and upload them to PyPI but these libraries raise errors when running pip install afterwards.)

• Automatically build wheels (and a source distribution) and upload to pypi when commits are tagged.
• Automatically build Anaconda dev releases when commits are not tagged.

Tagged releases for anaconda should be built by conda-forge, and their robot will automatically notice new PyPI releases and create pull requests.

The following libraries need to be updated and can have a single, version/OS independent wheel:

https://github.com/brightway-lca/brightway2-data https://github.com/brightway-lca/brightway2-io https://github.com/brightway-lca/brightway2-analyzer

This library needs to be updated, but needs version/OS-specific wheels:

https://github.com/brightway-lca/brightway2-calc

See https://github.com/brightway-lca/brightway2-speedups/blob/master/.github/workflows/build_wheels.yaml for an example of such wheels. The calc library relies on linear algebra libraries which are compiled separately for each version and OS. It should have pypardiso as a dependency, but note that no wheel for ARM Macs is available, as this can't use pypardiso, but instead should use scikit-umfpack, but it doesn't have binary distributions (https://pypi.org/project/scikit-umfpack/0.3.3/).

[cmutel]

Note the following changes to the original plan:

• Not using napoleon for documentation, just using numpy docstrings
• New beta documentation is online with source here, and is managed as a separate project
• Even if pre-commit hooks are in the template, they aren't necessarily being used, so we should also run these checks on commits