search

edgi-govdata-archiving / web-monitoring-ops

Documentation and configuration files for EDGI’s deployment of Web Monitoring tools.
GNU General Public License v3.0
1 stars 1 forks source link

Need instructions for updating Kops & Kubernetes #38

Closed Mr0grog closed 3 years ago

Mr0grog commented 4 years ago

Over in #35 I am rebuilding our Kubernetes cluster in AWS with a newer version of Kops and Kubernetes. Doing this is complicated because we’ve failed to keep both Kops and Kubernetes up to date, and what we previously had was no longer supported by current versions of the tools.

Kops provides tooling for upgrades: https://kops.sigs.k8s.io/operations/updates_and_upgrades/#upgrading-kubernetes

We need to document using it (which may be as simple as linking to the above) in the new runbook docs @lightandluck started to put together. This should be something we do pretty regularly (maybe once every 1-3 months?), since Kubernetes tends to do minor releases just about every 3 months and only the last 3 minor releases are generally supported.

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in seven days if no further activity occurs. If it should not be closed, please comment! Thank you for your contributions.

Mr0grog commented 3 years ago

Speaking of… we are overdue for an upgrade. I had wanted to upgrade once a quarter, but now it's been ~6 months.

This is a good opportunity to write a guide while I do it. :)

Mr0grog commented 3 years ago

Wound up doing another upgrade today for the first time in probably 9 months, which was clearly way too long. Things did not go well. :|

The biggest lesson here is don’t skip major version updates. Once I got everything successfully rolled back to the starting point (surprised I was actually able to do that), re-doing the upgrade version-by-version (very slow, but oh well) went pretty smooth, and really was just as simple as the instructions at https://kops.sigs.k8s.io/operations/updates_and_upgrades/#upgrading-kubernetes