GitHub Packaging Automation

NeonDaniel commented 2 years ago

What do we want to automate regarding versioning and publication?

For Neon packages, I've been satisfied with the workflows here: https://github.com/NeonGeckoCom/neon_mq_connector/tree/master/.github/workflows

On push to dev, increment alpha version and push to pypi
On push to master, publish release and push to pypi (assumes there was a manual version bump to a non-alpha which is indirectly enforced via branch protection)

If we don't want a dev/alpha branch, I would propose a version bump/alpha publish on any push to 'master', or possibly publish on release and leave alpha builds as manual.

NeonDaniel commented 2 years ago

Tooling can't increment the version number without requiring the dev to then pull the branch they just pushed

I think that if we just let automation take care of alpha versioning and leave manual version bumps to standalone PR's, it's reasonable to just never touch version.py during most PR's and any automated changes shouldn't conflict

ChanceNCounter commented 2 years ago

Indeed, and so long as CI refrains from altering PR branches, it should be clean.

The per-commit manual bump has been serving me well for pip and setup.py purposes, but it would be challenging to automate. That's what I'm more concerned about.

I think a dev or "nightly" branch is a good idea. Best practice in OSM's case, imo, would be for dev to reflect a full alpha --> release --> post cycle, whereas master only gets releases and post updates. Publish prerelease versions off of dev, and publish gold versions off master. Though master is always showing a previous state from the dev branch, for publishing, never the twain shall meet.

NeonDaniel commented 2 years ago

I agree with the dev/master structure, dev reflects changes that pass unit tests but otherwise are untested and master is always a production-ready release. For commit history, master is always a subset of dev that way.

For versioning per-commit, that would require a rebase before going to dev but I think it's worth it.. once it's pushed, incrementing makes sure PyPI always gets a unique version. Should be easy to resolve anyways (could even be done on GitHub)

NeonDaniel commented 2 years ago

Couple notes on the automated version updates and branch rules:

the action will fail to push to the base (master) branch if there are pretty much any branch protection rules (it's an unsigned, unreviewed commit).
the action will fail to push to the head (feature) branch if it's coming from an outside repo (runner runs in the same repo as the base branch if it's on PR).

ChanceNCounter commented 2 years ago

Had not previously occurred to me to do it in Python rather than in CI:

https://github.com/python-versioneer/python-versioneer

NeonDaniel commented 2 years ago

Had not previously occurred to me to do it in Python rather than in CI:

https://github.com/python-versioneer/python-versioneer

Same.. I'm more comfortable in GH Actions than Python packaging tools, but I think that could be another option.. Seems like it would still have to commit a modified file though if it's changing the version string?

ChanceNCounter commented 2 years ago

I haven't investigated too thoroughly, but my understanding is that the version string is calculated at runtime from the state of VCS, and for the rest,

To allow setup.py to compute a version too, a versioneer.py is added to the top level of your source tree, next to setup.py and the setup.cfg that configures it. This overrides several distutils/setuptools commands to compute the version when invoked, and changes setup.py build and setup.py sdist to replace _version.py with a small static file that contains just the generated version data.

NeonDaniel commented 2 years ago

changes setup.py build and setup.py sdist to replace _version.py with a small static file

I think this would basically come down to whether this _version.py change is reflected in git or only used in the PyPI package. I like having it in git so every git tag has an accurate version specifier

ChanceNCounter commented 2 years ago

Could the GH Action run the Versioneer build, get the version from the build, and derive the tag from that?

NeonDaniel commented 2 years ago

same commit issue if we want to update a version file or version string in the repo, but the tag could be done with that output; that's actually how I handle releases in Neon repos:

jobs:
  tag_release:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Get Version
        run: |
          VERSION=$(python setup.py --version)
          echo "VERSION=${VERSION}" >> $GITHUB_ENV
      - uses: ncipollo/release-action@v1
        with:
          token: ${{secrets.GITHUB_TOKEN}}
          tag: ${{env.VERSION}}

ChanceNCounter commented 2 years ago

Unless I've completely misunderstood, the version file in the repo is likewise dynamic. When you check the version from code, if you're working off source, it computes a version and build number from VCS. If you build the package, the version file is replaced with a hardcoded version string, but that doesn't need to be committed to VCS. That's just for the package itself.

NeonDaniel commented 2 years ago

Unless I've completely misunderstood, the version file in the repo is likewise dynamic. When you check the version from code, if you're working off source, it computes a version and build number from VCS. If you build the package, the version file is replaced with a hardcoded version string, but that doesn't need to be committed to VCS. That's just for the package itself.

I think that is correct, but ideally the version should be mapped 1:1 with a particular commit/package contents so those should always match.

The way I think of it is that version.py is the source of truth that setup.py reads from and that we update (manually or via automation)

OpenVoiceOS / ovos_skill_manager

GitHub Packaging Automation #74