Migrate stored version of K8s objects after upgrade or downgrade

[gdemonet]

Component: kubernetes

Why this is needed:

Today it is possible to create API objects (e.g., HPAs) in one version of Kubernetes, go through multiple upgrade cycles without touching those objects, and eventually arrive at a version of Kubernetes that can’t interpret the stored resource and crashes. See k8s.io/pr/52185.

See:

• Storage migration KEP
• StorageVersionHash KEP
• A migration controller that implements the automatic storage migration KEP

What should be done: Ensure all objects are stored in the "latest" version before and after any upgrade or downgrade.

Implementation proposal (strongly recommended):

Since we control the upgrade/downgrade of the cluster ourselves, it may not make sense to have an automatic migration in place. For the logic, get inspiration from the linked KEPs and controller. If StorageVersionHash is not available (before K8s 1.14), use the "old" approach to "get" + "replace" all objects.

Test plan:

• Have an automatic check that validates the migration tooling independently (create some object in an old version, run migration, check new version) - Note: would make sense to wait for StorageVersionHash to be available before implementing this
• Watch an object's stored apiVersion during an upgrade (observe two version switches, if possible)

[NicolasT]

https://github.com/kubernetes-sigs/kube-storage-version-migrator could be of use to get this done.