search

allenporter / k8s-gitops

Flux/Gitops managed k8s cluster
33 stars 1 forks source link

Ceph upgrade failed from 1.7.8, 1.7.9 `ReconciliationFailure` #432

Closed allenporter closed 2 years ago

allenporter commented 2 years ago

Alert ReconciliationFailure fired after https://github.com/allenporter/k8s-gitops/commit/07c83f50229b373a3598c34b04adbd036d837615

$ flux get hr -n rook-ceph
NAME            READY   MESSAGE                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         REVISION        SUSPENDED
rook-ceph       False   Helm upgrade failed: rendered manifests contain a resource that already exists. Unable to continue with update: PodSecurityPolicy "00-rook-privileged" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "rook-ceph"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "rook-ceph"   v1.7.9          False
allenporter commented 2 years ago

Oh actually this is failing on 1.7.9

$ helm history rook-ceph -n rook-ceph
REVISION        UPDATED                         STATUS          CHART           APP VERSION     DESCRIPTION
9               Thu Aug  5 01:07:16 2021        superseded      rook-ceph-v1.7.0                Upgrade complete
10              Thu Aug 12 23:25:38 2021        superseded      rook-ceph-v1.7.1                Upgrade complete
11              Fri Aug 27 03:30:28 2021        superseded      rook-ceph-v1.7.2                Upgrade complete
12              Fri Sep 10 02:05:22 2021        superseded      rook-ceph-v1.7.3                Upgrade complete
13              Fri Sep 24 20:08:59 2021        superseded      rook-ceph-v1.7.4                Upgrade complete
14              Fri Oct  8 00:21:01 2021        superseded      rook-ceph-v1.7.5                Upgrade complete
15              Wed Oct 20 19:52:38 2021        superseded      rook-ceph-v1.7.6                Upgrade complete
16              Fri Nov  5 02:32:18 2021        superseded      rook-ceph-v1.7.7v1.7.7          Upgrade complete
17              Fri Nov 19 02:00:53 2021        superseded      rook-ceph-v1.7.8v1.7.8          Upgrade complete
18              Fri Dec  3 01:13:44 2021        deployed        rook-ceph-v1.7.9v1.7.9          Upgrade complete
$ helm rollback rook-ceph -n rook-ceph 17
W1209 21:09:17.783304 1249920 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1209 21:09:17.785743 1249920 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1209 21:09:17.790284 1249920 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
Rollback was a success! Happy Helming!
allenporter commented 2 years ago

Following guidance in https://stackoverflow.com/questions/62964532/helm-not-creating-the-resources it makes sense to delete the unmanaged resource and let it get overwritten

$ kubectl delete podsecuritypolicy/00-rook-privileged
$ flux reconcile hr rook-ceph -n rook-ceph
► annotating HelmRelease rook-ceph in rook-ceph namespace
✔ HelmRelease annotated
◎ waiting for HelmRelease reconciliation
✔ applied revision v1.8.0