Automatic Cluster Upgrades

gianfranco-l commented 1 year ago

User Story

As a Giant Swarm Customer, I would like my clusters to be up to date so I can benefit from all new features and security patches
As a Giant Swarm Admin, I would like the cluster upgrades to be automated, so I can reduce the manual efforts when upgrading lots of clusters

Acceptance Criteria

[ ] Be able to define a schedule when clusters will be rolled automatically (staging Mondays, prod Fridays)
[ ] It should work with and without Gitops. In the case of GitOps, a PR is automatically raised (in the customer repo)

(More requirements in the RFC)

Context

Some work that has been done in the past on the topic:

Progress status as of Nov 22nd, 2022: this is an x-team between KaaS and RelEng. There is no capacity to work on this atm in most KaaS and RelEng teams. Rocket will do some steps towards its implementation (as there is some interest in doing this from a customer), consulting Honeybadger about possible solutions and ways to move forward (to keep the consistency of different use cases on the product)

gianfranco-l commented 1 year ago

added to horizon and roadmap repo + x-team label as agreed cc @puja108

teemow commented 1 year ago

@gawertm can you add some sub-issues here to create an overview of what Rocket is working on or planning to?

gawertm commented 1 year ago

thats what we still need to refine. discussions are pending for the final as well as intermediate solution. once done, I will refine :)

pipo02mix commented 1 year ago

I have added some details to the RFC noted after our meeting. IMO we should use this RFC to discuss and drive our decissions

teemow commented 1 year ago

@puja108 please write down the general direction we want to take with upgrades from a product point of view.

nprokopic commented 1 year ago

Similarly to this comment, the "Release channels and staged deployments" RFC is related to this.

As I commented in the other comment linked above, I will add shorter RFC overview, address and reply to comments there, clarify more, etc, and then I'll also post an update here, but a quick TL;DR here for how the RFC relates to automated cluster upgrades (it doesn't care about the cluster being WC or MC, it's for all):

There are multiple release channels, Test, Staging, Production (few different Production release channels, Edge/Current/LTS, but not getting into the details here)
Every new app version goes though all release channels, it starts in Test, then it gets promoted to Staging, then to production.
- The promotion is a fully automated process.
For every cluster it's specified which release channel it is using.
In the happy path, as soon as there is a new app/k8s/OS version in the release channel, the cluster gets that new version automatically. This delivery is fully automated process.
- This happy path can be overridden (so app/k8s/OS upgrade is postponed/deferred). This part is still not described in the RFC.
- But important, automated upgrades are the default, and you have to explicitly postpone the upgrade.

giantswarm / roadmap