Versions of apps and components beyond the first CAPI release

AKA feature development work after the migration.

Towards https://github.com/giantswarm/roadmap/issues/3057

User Story

As a Giant Swarm engineer, I want to release new minor and major versions of apps that my team owns and include that version in a new CAPI/CAPA release, so that we can deliver new features to our customers.
As a Team Turtles engineer, I want to release new minor versions of Kubernetes and include that version in a new CAPI/CAPA release, so that we can deliver new Kubernetes features to our customers.
As a cluster operator, I want to use new versions of apps and components (including Kubernetes), so that I use up-to-date versions and make use of new features that are released in those new versions.

Acceptance Criteria

[ ] Given my team owns an app which is a part of the CAPI/CAPA release, when my team releases a new minor or major version of an app, then we have to know without any doubt in which new CAPI release that new app version can be included.
[ ] Given my team owns or relies on some Kubernetes feature, when there is a new minor or patch version of Kubernetes, then we have to know without any doubt in which new CAPI/CAPA release that new Kubernetes version can be included.

Dependencies

[x] https://github.com/giantswarm/roadmap/issues/3321

Tasks

[ ] Define how we create new releases for delivering new versions of apps, i.e. for a new minor or a new major app release, define in which new release we can include new app version.
[ ] Define how we create new releases for delivering new versions of Kubernetes, i.e. for a new Kubernetes release, define in which new release we can include new Kubernetes version.
[ ] Define the above 2 just for the first 2-4 CAPI releases, so we cover the migration and the initial development for the next 3-6 months. Then we will re-evaluate the process and decide how to proceed.

Proposal

TL;DR:

v20 - the latest vintage release
- Kubernetes v1.25 (EOL)
- We only create new patch releases in case of critical bugs and security issues.
v25 - the first CAPI release, the migration landing zone (example version number for the sake of example scenario here, not a decided number)
- Kubernetes v1.25 (EOL)
- App versions are always exactly the same as in v20.
v26 - the second-ish CAPI release, the migration extension
- Kubernetes v1.26 (EOL) - the only change compared to v25
- App versions are always exactly the same as in v25.
- We upgrade clusters to v26 immediately after the migration.
v27 - the next feature development release
- Kubernetes v1.27 (EOL in June-July 2024)
- New app/component versions
v28 - around June-July when Kubernetes v1.27 will be EOL
- Kubenetes v1.28
- New app/component versions

Next - deep dive.

Let’s start with the next Kubernetes version.

New Kubernetes minor versions are always delivered in a new major Giant Swarm releases, and a new major Giant Swarm release always delivers a new Kubernetes version. So there is a 1-on-1 relation between Giant Swarm major and Kubernetes minor (we do this at least for the 4 CAPI releases in this proposal and then we evaluate if we can continue doing this).
Giant Swarm major version number matches the Kubernetes minor version number, e.g. Giant Swarm release v27 has Kubernetes v1.27.
- This reduces the cognitive load significantly.
- Doing both the migration and the upgrades right, and getting to use newer Kubernetes versions will require as much clarity and predictability as possible, and having a clear picture of app/component versions and what goes where, so this helps here.
After migrating to Giant Swarm v25 (the first CAPI release), we almost immediately upgrade to v26, as the extension of the migration process, or at least in the same day/week in case of some issues during/after the migration. Why do this upgrade immediately:
- We are quite behind the latest Kubernetes version v1.29, so we should catch up as soon as possible. This is the first step.
- The Kubernetes version that we use, v1.25, is already EOL.
- The Kubernetes v1.26 is also already EOL.
- We should not have an EOL version as the latest version we offer.
- It should really not be acceptable to not have even older EOL version as the latest version we offer.
- Kubernetes v1.26 should not bring any breaking changes. The most impactful one is the change in container image registry, which should not impact us.
So v26 is a special major release, like an extension of the migration.
When compared, v25 and v26 have the same app versions, and the only difference is the Kubernetes version. Reasoning here for keeping apps on the same versions:
- Clusters running on v25 will be freshly migrated from vintage.
- We want to upgrade them to a newer Kubernetes as soon as possible (see above why), but we also want to be careful and minimise the risks when upgrading Kubernetes right after the migration.
- Migration also handles the migration of apps and app configs. Keeping apps on same versions guarantees that all customers’ app configs are still valid, which is very much desirable if we want to upgrade to v26 almost immediately after the migration.
- We will have to support vintage v20 and CAPI/CAPA v25 and v26 as long as we have workload clusters to migrate from vintage to CAPA. And this, combined with v26 in the mix, also means that when we release a new patch version of an app in the latest vintage release v20, we have to release a new v25 and v26 patch release as well. The number of releases increases here. Having the same app versions on all of v20, v25 and v26 reduces the number of different app versions that we have to support.
New Kubernetes v1.26 patch releases, if they happen, are released in new v26 minor releases (I don’t expect that there will be new v1.25 releases).

Now about even newer Kubernetes and the new app versions.

The first CAPI release that we will use for the continuous feature development is v27.
We deliver Kubernetes v1.27 in our v27 release.
We deliver new major and minor app versions in our v27 release.
- We can have new major app versions in v27.0.0.
- Subsequent v27 minor and patch releases can deliver new minor and patch versions of apps.
- Subsequent v27 minor and patch releases cannot deliver new major versions of apps (those will go into v28).
We (roughly) define for how long we support v27.
Upgrade from v26 to v27 is a regular cluster upgrade and is planned and executed like other cluster upgrades until now. Ideally this happens as soon as possible while the v20->v25->v26 migrate & upgrade momentum is still strong.

Other notes:

The EOL for Kubernetes v1.27 is targeted for the end of June, so we should target to release v28 with v1.28 before that.
Once v28 is released, v27 would continue receiving only new patches, and the new development would continue on v28 only.

The following sections describe few scenarios.

Scenario 1: New releases for new patch versions of apps and components

This scenario shows how we:

Deliver new app patches to multiple releases.
Keep in app versions in-sync on v20 (the latest vintage) and v25 and v26.

Given we have these vintage (AWS) and CAPA releases:

AWS release v20.0.0
- Kubernetes v1.25
- MyApp v1.3.0
CAPA release v25.0.0
- Kubernetes v1.25
- MyApp v1.3.0
CAPA release v26.0.0
- Kubernetes v1.26
- MyApp v1.3.0

When we release new patch version of MyApp v1.3.1 that does not have any braking changes for customers

Then we have to create these new patch vintage (AWS) and CAPI (CAPA) releases:

AWS release v20.0.1
- Kubernetes v1.25
- MyApp v1.3.1
CAPA release v25.0.1
- Kubernetes v1.25
- MyApp v1.3.1
CAPA release v26.0.1
- Kubernetes v1.26
- MyApp v1.3.1

In case v27.0.0 also has MyApp v1.3.0, then we also create:

CAPA release v27.0.1
- Kubernetes v1.27
- MyApp v1.3.1

And the following upgrade and migration paths are then supported:

For vintage clusters that do not use the latest vintage release version:
- Upgrade from the latest AWS v19.x.y to AWS v20.0.1 release.
- Upgrade from v20.0.0 to v20.0.1 release.
For vintage clusters that use the latest vintage release version v20.0.1:
- Migration from AWS v20.0.1 to CAPA v25.0.1 release.
For CAPI clusters that have been already migrated to CAPA release v25.0.0 or newer:
- Upgrade from old v25.0.0 to new v25.0.1 release
- Upgrade from old v25.0.0 to new v26.0.1 release
- Upgrade from new v25.0.1 to new v26.0.1 release
- Upgrade from old v26.0.0 to new v26.0.1 release
- Upgrade from old v26.0.0 to new v27.0.1 release
- Upgrade from new v26.0.1 to new v27.0.1 release

Scenario 2: k8s 1.27 and new app minor and major versions

This scenario shows how we:

Deliver new app minor and major versions to the next feature development release.
Deliver new Kubernetes version.

Given we have these vintage (AWS) and CAPA releases:

AWS release v20.0.1
- Kubernetes v1.25
- FooApp v1.3.1
- BarApp v2.1.0
CAPA release v25.0.1
- Kubernetes v1.25
- FooApp v1.3.1
- BarApp v2.1.0
CAPA release v26.0.1
- Kubernetes v1.26
- FooApp v1.3.1
- BarApp v2.1.0

When we release a new major version of BarApp v3.0.0 and a new minor version of FooApp v1.4.0

Then we have to create a new major CAPI (CAPA) release v27 with new Kubernetes version v1.27 and new app versions (previous major versions do not get new patch nor minor versions).

AWS release v20.0.1 (unchanged)
- Kubernetes v1.25
- FooApp v1.3.1
- BarApp v2.1.0
CAPA release v25.0.1 (unchanged)
- Kubernetes v1.25
- FooApp v1.3.1
- BarApp v2.1.0
CAPA release v26.0.1 (unchanged)
- Kubernetes v1.26
- FooApp v1.3.1
- BarApp v2.1.0
CAPA release v27.0.0 (new)
- Kubernetes v1.27
- FooApp v1.4.0
- BarApp v3.0.0

Note: in rare cases new FooApp version v1.4.0 can be delivered also in v20, v25, v26, if and only if it contains fixes for critical issues or CVEs. In that case we would have additional following releases with FooApp v1.4.0:

AWS release v20.1.0
CAPA release v25.1.0
CAPA release v26.1.0

And the following upgrade and migration paths are then supported:

For vintage clusters that do not use the latest vintage release version:
- Upgrade from the latest AWS v19.x.y to AWS v20.1.0 release.
- Upgrade from v20.0.1 to v20.1.0 release.
For vintage clusters that use the latest vintage release version v20.1.0:
- Migration from AWS v20.1.0 to CAPA v25.1.0 release.
For CAPI clusters that have been already migrated to CAPA release v25.0.0 or newer:
- Upgrade from old v25.0.1 to new v25.1.0 release
- Upgrade from old v25.0.1 to new v26.1.0 release
- Upgrade from new v25.1.0 to new v26.1.0 release
- Upgrade from old v26.0.1 to new v26.1.0 release
- Upgrade from old v26.0.1 to new v27.0.0 release
- Upgrade from new v26.1.0 to new v27.0.0 release

I will add more comments about the development and testing.

giantswarm / roadmap