Refactor deployment workflows based on GitHub Releases

[lottspot]

This change set heavily refactors the automated deployment workflow to circumvent challenges with using GitHub actions for CD and to more closely match the current development release process. The workflow implemented is as follows:

• When a GitHub release is created, build a container image from the release and publish the image to GitHub packages
  • The main implication here is that we are performing builds when a release is created, rather than when a new tag is pushed
• After image is published, edit the k8s/app.yaml manifest to deploy the new image version, add the manifest to a zip file, and publish the zip file to the Release
  • For prod releases, the app.yaml manifest will be packaged with the manifests in k8s.prod
  • For pre-releases, the app.yaml manifest will be packaged with the manifests in k8s.preprod
• Next step is dependent on wether the release is marked as a pre-release or a full release:
  • For pre-releases, download the manifest files published in step 2 and apply them to the preprod cluster
  • For full releases, download the manifeast files published in step 2, acquire a diff via kubectl diff, and open a PR from develop into releases/v1 containing the diff in the description
• FULL RELEASES ONLY: Once the PR is merged, download the manifest files which were published to the Release and apply them to the production cluster

This workflow will allow the development team to perform releases without any intervention from the ops unit. Working examples of workflow steps:

• Published pre-release
• Pre-release workflow run
• Published full release
• Full release workflow run
• PR opened by full release workflow
• Deploy workflow from PR merge

[lottspot]

Note on changes to tagging conventions:

This PR sets the repository tag to :latest for the chime container image. Each new prod release will be pushed with a :latest tag in addition to its version tag. This will allow the repo version to safely remain labeled :latest, while still revving the active tag version on the cluster with each deploy.

[themightychris]

@lottspot incredible work! The only thing that could make this better is publishing deployment records (you could use deployments and their events to mediate the workflow, but that might be more trouble than it's worth. I think just creating the records against the ref being deployed and updating its status throughout the workflow will give us deployment status in the PR GUI and a sep slack event)

Edit: this one might be more appropriate: https://github.com/unacast/actions-github-deployment-status

[lottspot]

Deployments API integration has been added in f17c5d3. Working examples:

• Preprod release & deployment
• Prod release
• Prod deployment

And the resulting deployment objects

[MooseQuest]

I reread this with a clearer mind -- One that I like is the fact we are building the github package, making it available to pull done. This is excellent indeed.