search

kubernetes-sigs / secrets-store-csi-driver

Secrets Store CSI driver for Kubernetes secrets - Integrates secrets stores with Kubernetes via a CSI volume.
https://secrets-store-csi-driver.sigs.k8s.io/
Apache License 2.0
1.29k stars 296 forks source link

Upgrade-crds pod is erroring when attempting to validate crd apis #1634

Open Mo0rBy opened 2 months ago

Mo0rBy commented 2 months ago

What steps did you take and what happened: Tried to install secrets-store-csi-driver Helm chart. 1st pod I see being deployed is secrets-store-csi-driver-upgrade-crds-zkdc2 with a crds-upgrade container. The crds-upgrade container instantly fails with the following error:

crds-upgrade error validating "crds/secrets-store.csi.x-k8s.io_secretproviderclasses.yaml": error validating data: failed to download openapi: Get "https://172.20.0.1:443/openapi/v2?timeout=32s": dial tcp 172.20.0.1:443: connect: connection refused; if you choose to ignore these errors, turn validation off with --validate=false
crds-upgrade error validating "crds/secrets-store.csi.x-k8s.io_secretproviderclasspodstatuses.yaml": error validating data: failed to download openapi: Get "https://172.20.0.1:443/openapi/v2?timeout=32s": dial tcp 172.20.0.1:443: connect: connection refused; if you choose to ignore these errors, turn validation off with --validate=false
Stream closed EOF for cluster-addons/secrets-store-csi-driver-upgrade-crds-tzshc (crds-upgrade)

What did you expect to happen: I expect the installation to happen with no issues (mainly for the crds-upgrade pod to be able to pull the relevant data and continue its process).

Anything else you would like to add: I'm not actually sure where this crds-upgrade pod is coming from, as the Helm chart does not show any Deployment or even a Pod yaml for it, but I'm sure it's for this installation as it refreshes whenever I try to re-install and the error message fits.

Which provider are you using: AWS, but this is installing the base package, not AWS's additional package to interact with Secrets Manager

Environment:

Mo0rBy commented 2 months ago

Done some more looking and found that this pod is being created by a Job and the correct image tag for this container is registry.k8s.io/csi-secrets-store/driver-crds:v1.4.5, NOT what I put in the original post.

I upgraded from 1.4.4 to 1.4.5 to try and see of that would fix this, so I get the same behaviour with image tag 1.4.4

Mo0rBy commented 2 months ago

I have found a similar issue, but unfortunately it went stale and so there is no documented solution within the thread.

I have the ClusterRole described in that thread:

Screenshot 2024-09-08 at 13 53 41
Mo0rBy commented 2 months ago

I've just done a quick check with kubectl api-resources and we already have the CRD's available from the previous versions installation:

❯ kubectl api-resources | rg secrets-store
secretproviderclasses                            secrets-store.csi.x-k8s.io/v1            true         SecretProviderClass
secretproviderclasspodstatuses                   secrets-store.csi.x-k8s.io/v1            true         SecretProviderClassPodStatus
Mo0rBy commented 2 months ago

I've used a temporary solution of disabling the Helm install hook, so my installation does not wait for the install hook to pass first.

This means I have the failing Job + Pod for upgrade-crds still, but I dont care because I have all the actual secrets-store-csi-river DaemonSet pods so I have what I need.

Will leave this issue open as the crds-upgrade pod should still be working.