search

emissary-ingress / emissary

open source Kubernetes-native API gateway for microservices built on the Envoy Proxy
https://www.getambassador.io
Apache License 2.0
4.35k stars 681 forks source link

Helm chart does not clean-up on uninstall #3299

Open Atharex opened 3 years ago

Atharex commented 3 years ago

When I issue a delete of the Ambassador helm chart (v6.5.13), it times out

helm delete -n api-gateway ambassador
Error: timed out waiting for the condition

Turns out it spins up a pod to clean up CRDs. The problem is the service account of that pod does not have the proper permissions to perform the clean ups.

kubectl logs -f -n api-gateway ambassador-crd-cleanup-hxnt2
Error from server (Forbidden): customresourcedefinitions.apiextensions.k8s.io "devportals.getambassador.io" is forbidden: User "system:serviceaccount:api-gateway:ambassador" cannot delete resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope
Error from server (Forbidden): customresourcedefinitions.apiextensions.k8s.io "projectcontrollers.getambassador.io" is forbidden: User "system:serviceaccount:api-gateway:ambassador" cannot delete resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope
Error from server (Forbidden): customresourcedefinitions.apiextensions.k8s.io "projectrevisions.getambassador.io" is forbidden: User "system:serviceaccount:api-gateway:ambassador" cannot delete resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope
Error from server (Forbidden): customresourcedefinitions.apiextensions.k8s.io "projects.getambassador.io" is forbidden: User "system:serviceaccount:api-gateway:ambassador" cannot delete resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope

These seem to be the latest CRDs pertaining to the Projects feature. Please add proper permission for their clean-up!

Atharex commented 3 years ago

I don't think there is an easy way around this issue with the current implementation. Is there any other way to perform the cleanup? Or to add imagepullsecret handling also to this cleanup job?

>>> helm delete -n api-gateway ambassador
W0203 17:56:52.715895   27128 warnings.go:67] rbac.authorization.k8s.io/v1beta1 ClusterRoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRoleBinding
W0203 17:56:52.747138   27128 warnings.go:67] rbac.authorization.k8s.io/v1beta1 ClusterRoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRoleBinding
W0203 17:56:52.875318   27128 warnings.go:67] rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole
W0203 17:56:52.875320   27128 warnings.go:67] rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole
W0203 17:57:00.391371   27128 warnings.go:67] rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole
W0203 17:57:02.893872   27128 warnings.go:67] rbac.authorization.k8s.io/v1beta1 ClusterRoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRoleBinding
<this command is run in a separate tab>

>>> kubectl get pods -n api-gateway
NAME                           READY   STATUS             RESTARTS   AGE
ambassador-crd-cleanup-pfc6h   0/1     ImagePullBackOff   0          22s
>>> kubectl describe pod -n api-gateway ambassador-crd-cleanup-pfc6h
Name:         ambassador-crd-cleanup-pfc6h
Namespace:    api-gateway
...
Events:
  Type     Reason     Age                      From               Message
  ----     ------     ----                     ----               -------
  Normal   Scheduled  35s                      default-scheduler  Successfully assigned api-gateway/ambassador-crd-cleanup-pfc6h to worker-02
  Normal   Pulling    14s (x2 over 32s)        kubelet            Pulling image "buoyantio/kubectl"
  Warning  Failed     12s (x2 over 30s)        kubelet            Failed to pull image "buoyantio/kubectl": rpc error: code = Unknown desc = Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
  Warning  Failed     12s (x2 over 30s)        kubelet            Error: ErrImagePull
  Normal   BackOff    <invalid> (x2 over 30s)  kubelet            Back-off pulling image "buoyantio/kubectl"
  Warning  Failed     <invalid> (x2 over 30s)  kubelet            Error: ImagePullBackOff