Assess upgrade readiness before running an upgrade

User Story

As a cluster admin, I want to verify easily that a cluster meets all requirements to endure an upgrade.

Details, Background

In our workload cluster upgrade docs we collect a number of best practices and recommendations to help ensure that workload clusters and workloads are prepared for a cluster upgrade.

There is derailed/popeye as a CLI tool to vet clusters against a number of rules. It is licensed under the Apache 2.0 licence. Users can configure it to mute or re-configure certain checks.

We have already contributed to popeye in order to perform the checks for what we consider vital before a workload cluster upgrade, and will continue to do so.

Checks to add

[x] Consider high-availability masters (a. k. a. control plane nodes) https://github.com/derailed/popeye/issues/136 https://github.com/derailed/popeye/pull/131
[x] Avoid pods without owner reference (e. g. ReplicaSet, DaemonSet, StatefulSet) https://github.com/derailed/popeye/issues/137 https://github.com/derailed/popeye/pull/132
[ ] Ensure redundancy for workloads
- [x] Services: https://github.com/derailed/popeye/issues/138, https://github.com/derailed/popeye/pull/135
[ ] Distribute workloads across nodes https://github.com/derailed/popeye/issues/143 https://github.com/derailed/popeye/pull/149
[ ] Handle termination signals in Pods
[ ] Avoid ephemeral resources
- [ ] emptyDir volumes
- [ ] CSI ephemeral volumes
- [ ] generic ephemeral volumes
[ ] Configure webhook timeouts
[ ] Check that the images needed are available in the registry
[ ] Make Docker images as small as possible

Long term perspective

If the checks provide useful when performed manually by Giant Swarm staff and customers before performing an upgrade, we can consider running popeye in the cluster and providing results in a standard way.

This can either happen as a permanent service, potentially as an app to be installed from a Giant Swarm catalog, or as a necessary step before an upgrade (executed via a core component / default app).

giantswarm / roadmap