Multipackage releases on CI

[lmmx]

Multipackages

• Use 'multipackages': multiple Python packages in one repo
  • Inspired by visidata apps (but I'm not sure that that actually ships them separately)

We typically just have a single setup.py script which uses find_packages("src") and follows a src layout.

To ship multiple packages (let's say for APIs named "a", "b", "c"), we'd want:

botodto-monorepo/
  botodto-a/
    setup.py
    botodto_a/
      __init__.py
      ...
  botodto-b/
    setup.py
    b/
      __init__.py
      ...
  botodto-c/
    setup.py
    c/
      __init__.py
      ...

Releases on CI

• Control package release via CI: only trigger a release when the code for an API changes.
  • Don't want to spam PyPI or have trivial releases. Every package release should have a good reason.

Discussion

name: Publish botodto-a to PyPI

on: 
  push:
    branches:
      - main
    paths:
      - 'botodto-a/**'
      - 'updated_packages.json'
      - '.github/workflows/botodto-a.yml'

jobs:
  build-n-publish:
    name: Build and publish botodto-a to PyPI
    runs-on: ubuntu-latest
    environment:
      name: pypi
      url: https://pypi.org/p/botodto-a
    permissions:
      id-token: write
    steps:
    - uses: actions/checkout@v3
    - name: Cache pip packages
      uses: actions/cache@v3
      with:
        path: ~/.cache/pip
        key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
        restore-keys: |
          ${{ runner.os }}-pip-
    - name: Check if package was updated
      id: check
      run: echo "::set-output name=updated::$(jq -r '.botodto-a' updated_packages.json)"
    - name: Set up Python
      if: steps.check.outputs.updated == 'true'
      uses: actions/setup-python@v2
      with:
        python-version: '3.x'
    - name: Install pypa/build
      if: steps.check.outputs.updated == 'true'
      run: >-
        python -m
        pip install
        build
        --user
    - name: Build a binary wheel and a source tarball
      if: steps.check.outputs.updated == 'true'
      run: >-
        python -m
        build
        --sdist
        --wheel
        --outdir dist/
    - name: Publish distribution to PyPI
      if: steps.check.outputs.updated == 'true' && startsWith(github.ref, 'refs/tags')
      uses: pypa/gh-action-pypi-publish@release/v1
      with:
        repository_url: https://test.pypi.org/legacy/

• This is a common pattern, similar to the one used in the datamodel-code-generator package