Roundup handling VERSION.txt

[nutjob4life]

🐛 Describe the bug

With the removal of Versioneer underway, we also need to address how the Roundup Action handles release/* branches for Python repositories (and for Java repositories).

Currently, Roundup does some juggling with branches and tags in order to trick Versioneer into generating the right release. Now that we're removing Versioneer and using VERSION.txt files, we'll need to change our approach. This can be done in the Roundup Action or in pds-github-util—whichever is more appropriate. Probably both.

We should also revisit how we want this all to work, as a discussion on 2021-09-23, there are some corner cases we may not have anticipated.

Merge to main

This should generate an unstable release which for Python means a publication of a package-1.2.3-dev-YYYYMMDD-py3.9.whl wheel to test.pypi.org¹ and removal of all existing -dev- releases from GitHub. The VERSION.txt file is read to create the 1.2.3 part of the wheel name but is otherwise untouched. pds-github-util takes care of dropping all previous -dev- releases and uploading the new asset package-1.2.3-dev-YYYYMMDD-py3.9.whl to GitHub.

¹It can be argued that test.pypi.org is not an appropriate place for a -dev- release. The site indicates it's for testing of release tools, not for hosting of snapshot/development packages.

Push branch release/1.2.4

Anyone creates and checks out release/1.2.4 in their local repository and pushes this with --set-upstream origin to GitHub. The stable workflow should then activate:

1. Actions checks out the release/1.2.4 branch
2. Roundup detects the branch name release/1.2.4 and writes 1.2.4 to VERSION.txt, overwriting whatever's there
3. Roundup executes bdist_wheel, creating package-1.2.4-py3.9.whl
4. Roundup commits this change to the release/1.2.4 branch; pushes the change²
5. Roundup creates annotation tag v1.2.4 and pushes it I don't think this is needed because pds-github-util's release-creation makes the tag too
6. Roundup uses Twine to upload wheel to pypi.org
7. pds-github-util creates the non-snapshot release in GitHub and uploads the package-1.2.4-py3.9.whl artifact to it

²Should it also merge this change into main? I kind of think so.

Push branch release/1.2

Previously, we had contrived to make this automatically detect the partial branch name and then auto-create a corresponding release/1.2.X branch where X was the highest Y + 1 from previous release/1.2.Y branches. However, this left am ambiguity: perhaps release/1.2.4 because we detected release/1.2.3—but what if there was a push to release/1.2.4 and a push to release/1.2 which had diverged? It makes sense to have a branch for each release for the backporting of bug fixes from main but not for the ambiguous release/1.2 branch.

To avoid that ambiguity and in the spirit of being explicit, maybe we should drop this idea.

What about the Release Engineer?

A part of my wants to balk at automating releases in this way because it does away with the role of a Release Engineer, the person who's expertise and job description is to make, but more importantly, sign releases. Fully automating stable releases means removing the digital chain of responsibility to an individual for ensuring the integrity of the software. Consider an insider-threat with access to GitHub who substitutes a crafted key and enables malicious software to be apparently be signed by the Engineering Node.

That's part of why I included this label on the key currently being used:

PDS Engineering Node (SIGNING KEY ONLY from automated processes; trust accordingly)

Apache and Plone (in my own experience) go through specific ceremonies for releases that may be onerous (voting, signing, etc.) but conceivably serve a higher purpose.

📚 Version of Software Used

roundup-action = 6dca7c7 pds-github-util = 0.25.3 pds-template-repo-python = 0.1.0

---

🦄 Related requirements

⚙️ Engineering Details

[jordanpadams]

@nutjob4life @tloubrieu-jpl @MJJoyce I am ok with doing away with stable releases from a GitHub action point of view this looks to have ballooned on scope to the point that it probably isn't worth it.

That being said, if we get rid of that, then I want a script that I can run on my laptop that can automate this. (Similar to that build-java.sh script but preferably a little smarter and we need 1 for python).

The whole point of doing the stable release was because manually incrementing a version.txt file and then manually running 19 steps to tag a release is extraordinarily painful when you need to tag 10 repos at delivery time. I would like a way to say "release software X with a new minor version", or "bug fix" or "major version" and voila, magic happens. We had this in the conman software back in the day, but we trashed it (deservedly) for the GitHub action. I'm ok going back to using something like pds-github-util from my local trusted environment to do this.

[nutjob4life]

Well hang on @jordanpadams! I think we've got a lot in place to support this. I just wanted to present a couple alternative points of view (big projects with dedicated release engineers and thousands of users where accountability matters). We have lots of little projects right now and only dozens of users.

I'm not throwing in the towel on stable-cicd.yaml yet!

But if we narrow the focus a bit (such as requiring you to branch release/1.2.3 and thus avoiding the case of what-to-do with multiple ambiguous branches) it might make things doable.

[jordanpadams]

i'm ok with that

[MJJoyce]

²Should it also merge this change into main? I kind of think so.

+1, should merge this back into main as well IMO

Push branch release/1.2

Previously, we had contrived to make this automatically detect the partial branch name and then auto-create a corresponding release/1.2.X branch where X was the highest Y + 1 from previous release/1.2.Y branches. However, this left am ambiguity: perhaps release/1.2.4 because we detected release/1.2.3—but what if there was a push to release/1.2.4 and a push to release/1.2 which had diverged? It makes sense to have a branch for each release for the backporting of bug fixes from main but not for the ambiguous release/1.2 branch.

To avoid that ambiguity and in the spirit of being explicit, maybe we should drop this idea.

I'm in favor of dropping the automatic handling. I'd rather the releases be explicit even if that makes it more manual. Semi-related, we should always use 1.2.0 over 1.2 IMO.

What about the Release Engineer?

A part of my wants to balk at automating releases in this way because it does away with the role of a Release Engineer, the person who's expertise and job description is to make, but more importantly, sign releases. Fully automating stable releases means removing the digital chain of responsibility to an individual for ensuring the integrity of the software. Consider an insider-threat with access to GitHub who substitutes a crafted key and enables malicious software to be apparently be signed by the Engineering Node.

That's part of why I included this label on the key currently being used:

PDS Engineering Node (SIGNING KEY ONLY from automated processes; trust accordingly)

What about making the stable build not run constantly and require the release engineer to pass additional information to the workflow when executing it manually? I realize it's not necessarily "signing" the release in any meaningful way but at least we can document who did the release at some level.

[MJJoyce]

Thoughts for when we tackle this.

_BuildStep currently pulls tag candidates via Git

candidate = invokeGIT(['describe', '--always', '--tags'])

For consistency should we instead leverage pds-github-util and the version fetching utilities that it provides?

[nutjob4life]

@MJJoyce yes.

(Either that, or roll all the release handling stuff in pds-github-util into the Roundup.)

[nutjob4life]

Update: this seems to be working correctly in the sandbox. Leaving this issue open here until we merge things back.