KVM Release v13.0.0 with Kubernetes v1.18

cornelius-keller commented 4 years ago

Kubernetes v1.18.x

https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-XXX.md

Provide a new release with the kubernetes version v1.18.x

[x] Check with the teams about wip changes within k8scloudconfig and the provider-specific operators. Decide with version folder to use.
[x] Retag new image in retagger
[x] Check upstream changelogs.
- [x] Check urgent upgrade notices.
- [x] Check deprecation notices.
- [x] Check component upgrades.
- [x] Calico latest: 3.16.1 / 3.15.3 - release: 3.15.3 use 3.16 if possible
- [x] ETCD latest: 3.14.3 - release: 3.14.3
- [x] Container Linux latest: 2605.5.0 - release: 2512.5.0 upgrade incompatible
[x] Create a PR in k8scloudconfig with the new kubernetes version
- [x] Update k8s version in the different template files
- [ ] Use k8s version for k8s-addons (e.g. https://github.com/giantswarm/k8scloudconfig/blob/4695b7d6eb35eae234d0ce0c0c09f4b412525a68/v_4_7_0/files/conf/k8s-addons#L5)
[x] Update cluster-autoscaler
[x] Vendor k8scloudconfig in the provider-specific operators
[x] Create a wip release (major version change if it is a minor kubernetes upgrade)
[x] Do this for each provider
[x] Check Prometheus rules for deprecated metrics and fix them if necessary

Check migration recommendations

Check migration recommendations from kubernetes and decide what we need to document for the customer and what we should migrate automatically for them

[x] Do we need to change our own resources within the tenant clusters?
[x] What do we need to communicate to our customers?
[x] Anything else that would be better automated for the upgrade?

Run e2e and conformance tests

[x] Start a new cluster and run conformance tests against it
[x] Start a new cluster and run e2e tests against it
[x] Do this for each provider

Check Core Components

[x] Check core components for updates and include them in cluster-operator (checked 2020-09-24)
- [x] Cert-exporter latest: 1.3.0 / release: 1.3.0
- [x] Chart-operator latest: 0.13.2 (helm 2) 2.3.2 (helm 3) / release: 0.13.2 ~* [ ] Cluster-autoscaler~
- [x] CoreDNS latest: 1.2.0 / release: 1.2.0
- [x] Kube-state-metrics latest: 1.2.0 / release: 1.1.1 todo: use new version
- [x] Net-exporter latest: 1.9.2 / release: 1.9.2
- [x] Nginx ingress controller
- [x] Node-exporter latest: 1.4.1 / release: 1.3.0 todo: use new version
[x] All helm releases in the giantswarm namespace should be in status deployed
[x] Core components are runnning without errors
- [x] Cert-exporter
- [x] Chart-operator
- [x] Cluster-autoscaler
- [x] CoreDNS
- [x] Kube-state-metrics
- [x] Net-exporter
- [x] Nginx ingress controller
- [x] Node-exporter
[x] Cluster-operator is running without errors
[x] Cluster-operator is able to create chart-configs in the tenant

Test migration (both cluster functionality itself and workloads)

[x] Start a cluster with an older version and upgrade the cluster to the new release
[x] Upgrade a cluster that is not critical but has some more real workloads (eg giantswarm website, dev cluster of customer)
[x] Do this for each provider

Write summary for the release and update docs

[ ] Write some release notes
[ ] Check if our docs need to be updated
[ ] Announce the release internally (SEs are important here)
[ ] Announce the release to the customers

stone-z commented 4 years ago

Began testing release defined in this PR. Reviewing components for release, two components were updated today which should be included:

Kube-state-metrics 1.2.0
Node-exporter 1.4.1

stone-z commented 4 years ago

Kube-state-metrics 1.2.0 and node-exporter 1.4.1 have been added to the release. They are internal changes to the app packaging only

brinker211 commented 4 years ago

Taking into KVM.

yulianedyalkova commented 3 years ago

There is an issue with kube-state-metrics that requires a manual delete of the deployment on upgrade. More context here.

Currently it's not possible to run cncf against our test installations because of the test clusters mentioned here: https://github.com/giantswarm/giantswarm/issues/14236

yulianedyalkova commented 3 years ago

So I tried running cncf 3 times in prow and got 3 different failures:

A test failed and then sonobuoy couldn't give info of what's happening:

PLUGIN     STATUS   RESULT   COUNT
e2e     failed   failed       1
systemd-logs   complete   passed       4

[2020-11-18T15:43:23Z] Results summary
timeout waiting for results

[2020-11-18T15:43:25Z] Sonobuoy logs
time="2020-11-18T15:43:35Z" level=error msg="could not create sonobuoy client: couldn't get sonobuoy api helper: could 
not get api group resources: Get \"https://api.fgd7h.k8s.gorgoth.gridscale.kvm.gigantic.io/api?timeout=32s\": net/http: TLS 
handshake timeout"

Coredns didn't come up within the timeout

Preflight checks failed

[run-tests : run-tests] [2020-11-19T14:08:40Z] Running cncf
[run-tests : run-tests] time="2020-11-19T14:08:40Z" level=error msg="Preflight checks failed"
[run-tests : run-tests] time="2020-11-19T14:08:40Z" level=error msg="namespace already exists"

I tried running cncf against the old release and it failed because coredns took 1 hour to come up (2. failure). The only app in the TC namespace was the chart-operator, the chart was installed within the TC but the pod itself was missing.

I'm also assuming that 1. and 3. have something to do with rfjh2 being updated at the same time. I ran cncf 3 times manually and they were all green (except the fact that coredns takes a long time to come up).

tfussell commented 3 years ago

I have a PR to fix the slow start of k8s-addons (which installs CoreDNS) here https://github.com/giantswarm/k8scloudconfig/pull/832 I'm testing it today.

yulianedyalkova commented 3 years ago

Just for trackability: the problem with coredns coming up is that chart-operator cannot get deployed because of:

Events:
  Type     Reason        Age                 From                   Message
  ----     ------        ----                ----                   -------
  Warning  FailedCreate  96s (x19 over 23m)  replicaset-controller  Error creating: pods "chart-operator-cd7957c78-" is forbidden: no PriorityClass with name giantswarm-critical was found

This eventually gets created after ~40 minutes and then everything is happy. It also happens for release v12.3.2 so it's not a regression of the new release.

MarcelMue commented 3 years ago

Just for trackability: the problem with coredns coming up is that chart-operator cannot get deployed because of:
Events:
  Type     Reason        Age                 From                   Message
  ----     ------        ----                ----                   -------
  Warning  FailedCreate  96s (x19 over 23m)  replicaset-controller  Error creating: pods "chart-operator-cd7957c78-" is forbidden: no PriorityClass with name giantswarm-critical was found
This eventually gets created after ~40 minutes and then everything is happy. It also happens for release v12.3.2 so it's not a regression of the new release.

Interesting - do we know where the prio class is coming from? IIRC it was in k8scloudconfig

yulianedyalkova commented 3 years ago

Yesterday I managed to follow it through k8scloudconfig -> CLOUD_CONFIG_PATH in k8s-kvm -> qemu_node_setup -> qemu. I didn't see where it actually gets created though, will spend some time on it later.

tfussell commented 3 years ago

The priority class is created in k8s-addons which is fixed in my k8scloudconfig PR. I'll merge it and update kvm-operator today.

giantswarm / roadmap