Design an InVEST automated release process

[phargogh]

The InVEST release process has been quite manual (documented here: https://bitbucket.org/natcap/invest/wiki/Release Checklist) and pretty involved, requiring a nontrivial amount of engineering time to become confident in our releases before we send it out.

In short, it would be a fantastic improvement to our team productivity if we could design a process that:

• Instills confidence in our releases by thoroughly testing the final executable. Related tasks for this are:
  • Testing of models via the InVEST user interface would be the most useful thing to do here
  • Building and testing the python packages directly across all supported architectures and versions
• Requires minimal (if any) human interactions in order to complete the release. Related tasks would be:
  • Tagging the UG repository with the correct revision found in Makefile (or possibly failing the build if the rev in Makefile is really out of date?)
  • Updating HISTORY.rst with the correct date
  • Automatically incrementing the InVEST release version number as part of the build process
  • Building the release, including:
    • Uploading artifacts to our storage bucket
    • Uploading python binaries to PyPI
  • Announcing the latest InVEST release to interested audiences via needed channels (email? slack? forums?)

Other related questions we’ll want to consider are:

• Whether the release process above should happen for all commits, or if a release should happen at most once per unit of time (once per day? once per week?)
• Should there be a separate branch (separate from develop) to indicate features that should be released via this automated release process?
• How will errors that come up during an automated release be handled?

[phargogh]

I've started this process by working on a couple of scripts to do some of the text processing required for this sort of automation, all written (at present) to ci/release/. Next steps are to figure out how to create a new release object in the repository via the github api during an actions run.

[phargogh]

I've added a script to ci/release/tag-github-repo.py` to tag a target github repository.

The best thing to do at this point will be to work up a prototype on a sample repository on my own account. This prototype should have two workflows:

1. A workflow that will do a 'build' (maybe zip something up or something like that). This should execute on every push, PR and Release. If the triggering event is a release, the build should also upload its artifact to the correct release in the correct repository.
2. A workflow that will create a github release, but only if there are new commits on master.
3. I'm concerned that all of this will cause release to be automatically created on forks as well. This prototype should only trigger the creation of releases IFF on the main repository, not on a fork.

[phargogh]

• [x] As an additional functional requirement, we should not proceed with a bugfix release if the builds/tests on master are failing.

[phargogh]

I'm currently having some challenges with ensuring that builds happen under the following circumstances:

• When commits are pushed to a branch
• When a release is created

Right now there's also a complication which is that pushing tags also counts as a push. Is this incorrect? Unlike with mercurial (where making a tag also makes a new commit), pushing a tag will also trigger a rebuild and retest.

• [x] So the thing to consider here is how to ensure that we're only creating the needed binaries once.

[dcdenu4]

Oh that's tricky. Yea I believe Git treats tags like branches basically, so you have to push them to the remote server like you would a local branch.

[phargogh]

@dcdenu4 it makes sense given how git works! But it's still a little strange to rebuild things for the same commit (just with a different version number).

[phargogh]

This is coming along! I've gotten the release-object and binaries automation up and running in my fork, but there's still some extra work to be done to streamline what is currently working.

• [x] Reduce duplication of build effort. Builds are currently executed twice in a release (once in response to the release, once in response to the push of tags), and they should only be built once. I suspect this will mean removing the response to release events and instead detecting if a relevant release object exists on a tag.
• [x] If builds are failing on master, we should not create a release, and we should instead send an alert to our slack channel notifying us that the build could not happen and why.
• [ ] If an automated release does happen, a PR should automatically be opened into any open release branches with some useful text about it being an automated release and what to do with the PR (it's a chance to review any incoming changes, if anything doesn't look right, a bugfix branch should be opened off of master and PRed into the release branch.)

[phargogh]

Before I leave this for the weekend, just making notes on where things were left off. Remaining tasks are:

• [x] Clean up python scripts that are no longer needed. A lot of the interactions with github have been replaced with calls to the hub CLI (which github actions plays very, very nicely with, btw), and so the python scripts I originally wrote for this can be mostly replaced with a little shell scripting, rendering the python scripts unnecessary.
• [x] Test this via cron on feature/automated-releases to make sure that the scheduled part of this works as expected.
• [x] Implement slack messages on success and failure
• [x] Current versioning does not handle pre-releases (alpha, beta, etc.). These should either be handled or an informative error should prevent us from accidentally creating an erroneous version.
• [x] Use pandoc or something similar to convert RST in HISTORY to MD for the text of releases.
• [x] Create a PR from master into each release branch once a new release object has been created. This should be subsumed by https://github.com/natcap/invest/issues/66

[phargogh]

It turns out that you cannot have events authenticated with $GITHUB_TOKEN trigger other events (see https://help.github.com/en/actions/reference/events-that-trigger-workflows). If we want to be able to trigger other events, it must be authenticated with a personal access token. There are a couple of possible ways that we could move forward from this

1. Continue with full release automation The cron action will need to be authenticated with a personal access token, but after doing so, we should then be able to have the job trigger the other binary builds as needed.
2. Automate everything except the release creation Instead of automating releases, we could determine by hand when to create the release object, and then have a pile of other automation start up in response to this. Fortunately, retagging the repo with the same commit can be done, and the release object will be updated to point to the correct revision.

Somewhat interestingly, these two options are not mutually exclusive. Let's say master is in a state where the full release automation won't do an automated release (like when tests are failing on master). We could still follow a few steps by hand to have the same effect:

1. Update the HISTORY on a branch
2. PR the branch into master
3. Once the PR is merged into master, create a release object with the appropriate text derived from HISTORY (and converted to Markdown, such as via pandoc).
4. Once the release is created, this will push a tag and will therefore trigger a build of the new ref. And because actions will be building at a ref, the files will be uploaded to the release object.

[phargogh]

Currently pending #83

[phargogh]

We now have an InVEST automated release process. I'll close this issue and instead work on improving the existing process.