Azure Key Vault provider for Secret Store CSI driver allows you to get secret contents stored in Azure Key Vault instance and use the Secret Store CSI driver interface to mount them into Kubernetes pods.
Error from server (Invalid): error when applying patch:
{"metadata":{"annotations":{"controller-gen.kubebuilder.io/version":"v0.4.0","kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apiextensions.k8s.io/v1\",\"kind\":\"CustomResourceDefinition\",\"metadata\":{\"annotations\":{\"controller-gen.kubebuilder.io/version\":\"v0.4.0\"},\"creationTimestamp\":null,\"name\":\"secretproviderclasses.secrets-store.csi.x-k8s.io\"},\"spec\":{\"group\":\"secrets-store.csi.x-k8s.io\",\"names\":{\"kind\":\"SecretProviderClass\",\"listKind\":\"SecretProviderClassList\",\"plural\":\"secretproviderclasses\",\"singular\":\"secretproviderclass\"},\"scope\":\"Namespaced\",\"versions\":[{\"name\":\"v1alpha1\",\"schema\":{\"openAPIV3Schema\":{\"description\":\"SecretProviderClass is the Schema for the secretproviderclasses API\",\"properties\":{\"apiVersion\":{\"description\":\"APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources\",\"type\":\"string\"},\"kind\":{\"description\":\"Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds\",\"type\":\"string\"},\"metadata\":{\"type\":\"object\"},\"spec\":{\"description\":\"SecretProviderClassSpec defines the desired state of SecretProviderClass\",\"properties\":{\"parameters\":{\"additionalProperties\":{\"type\":\"string\"},\"description\":\"Configuration for specific provider\",\"type\":\"object\"},\"provider\":{\"description\":\"Configuration for provider name\",\"type\":\"string\"},\"secretObjects\":{\"items\":{\"description\":\"SecretObject defines the desired state of synced K8s secret objects\",\"properties\":{\"annotations\":{\"additionalProperties\":{\"type\":\"string\"},\"description\":\"annotations of k8s secret object\",\"type\":\"object\"},\"data\":{\"items\":{\"description\":\"SecretObjectData defines the desired state of synced K8s secret object data\",\"properties\":{\"key\":{\"description\":\"data field to populate\",\"type\":\"string\"},\"objectName\":{\"description\":\"name of the object to sync\",\"type\":\"string\"}},\"type\":\"object\"},\"type\":\"array\"},\"labels\":{\"additionalProperties\":{\"type\":\"string\"},\"description\":\"labels of K8s secret object\",\"type\":\"object\"},\"secretName\":{\"description\":\"name of the K8s secret object\",\"type\":\"string\"},\"type\":{\"description\":\"type of K8s secret object\",\"type\":\"string\"}},\"type\":\"object\"},\"type\":\"array\"}},\"type\":\"object\"},\"status\":{\"description\":\"SecretProviderClassStatus defines the observed state of SecretProviderClass\",\"properties\":{\"byPod\":{\"items\":{\"description\":\"ByPodStatus defines the state of SecretProviderClass as seen by an individual controller\",\"properties\":{\"id\":{\"description\":\"id of the pod that wrote the status\",\"type\":\"string\"},\"namespace\":{\"description\":\"namespace of the pod that wrote the status\",\"type\":\"string\"}},\"type\":\"object\"},\"type\":\"array\"}},\"type\":\"object\"}},\"type\":\"object\"}},\"served\":true,\"storage\":true}]},\"status\":{\"acceptedNames\":{\"kind\":\"\",\"plural\":\"\"},\"conditions\":[],\"storedVersions\":[]}}\n"}},"spec":{"versions":[{"name":"v1alpha1","schema":{"openAPIV3Schema":{"description":"SecretProviderClass is the Schema for the secretproviderclasses API","properties":{"apiVersion":{"description":"APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources","type":"string"},"kind":{"description":"Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds","type":"string"},"metadata":{"type":"object"},"spec":{"description":"SecretProviderClassSpec defines the desired state of SecretProviderClass","properties":{"parameters":{"additionalProperties":{"type":"string"},"description":"Configuration for specific provider","type":"object"},"provider":{"description":"Configuration for provider name","type":"string"},"secretObjects":{"items":{"description":"SecretObject defines the desired state of synced K8s secret objects","properties":{"annotations":{"additionalProperties":{"type":"string"},"description":"annotations of k8s secret object","type":"object"},"data":{"items":{"description":"SecretObjectData defines the desired state of synced K8s secret object data","properties":{"key":{"description":"data field to populate","type":"string"},"objectName":{"description":"name of the object to sync","type":"string"}},"type":"object"},"type":"array"},"labels":{"additionalProperties":{"type":"string"},"description":"labels of K8s secret object","type":"object"},"secretName":{"description":"name of the K8s secret object","type":"string"},"type":{"description":"type of K8s secret object","type":"string"}},"type":"object"},"type":"array"}},"type":"object"},"status":{"description":"SecretProviderClassStatus defines the observed state of SecretProviderClass","properties":{"byPod":{"items":{"description":"ByPodStatus defines the state of SecretProviderClass as seen by an individual controller","properties":{"id":{"description":"id of the pod that wrote the status","type":"string"},"namespace":{"description":"namespace of the pod that wrote the status","type":"string"}},"type":"object"},"type":"array"}},"type":"object"}},"type":"object"}},"served":true,"storage":true}]},"status":{"acceptedNames":{"kind":"","plural":""},"conditions":[],"storedVersions":[]}}
to:
Resource: "apiextensions.k8s.io/v1, Resource=customresourcedefinitions", GroupVersionKind: "apiextensions.k8s.io/v1, Kind=CustomResourceDefinition"
Name: "secretproviderclasses.secrets-store.csi.x-k8s.io", Namespace: ""
for: "crds/secrets-store.csi.x-k8s.io_secretproviderclasses.yaml": CustomResourceDefinition.apiextensions.k8s.io "secretproviderclasses.secrets-store.csi.x-k8s.io" is invalid: status.storedVersions[1]: Invalid value: "v1": must appear in spec.versions
Error from server (Invalid): error when applying patch:
{"metadata":{"annotations":{"controller-gen.kubebuilder.io/version":"v0.4.0","kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apiextensions.k8s.io/v1\",\"kind\":\"CustomResourceDefinition\",\"metadata\":{\"annotations\":{\"controller-gen.kubebuilder.io/version\":\"v0.4.0\"},\"creationTimestamp\":null,\"name\":\"secretproviderclasspodstatuses.secrets-store.csi.x-k8s.io\"},\"spec\":{\"group\":\"secrets-store.csi.x-k8s.io\",\"names\":{\"kind\":\"SecretProviderClassPodStatus\",\"listKind\":\"SecretProviderClassPodStatusList\",\"plural\":\"secretproviderclasspodstatuses\",\"singular\":\"secretproviderclasspodstatus\"},\"scope\":\"Namespaced\",\"versions\":[{\"name\":\"v1alpha1\",\"schema\":{\"openAPIV3Schema\":{\"description\":\"SecretProviderClassPodStatus is the Schema for the secretproviderclassespodstatus API\",\"properties\":{\"apiVersion\":{\"description\":\"APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources\",\"type\":\"string\"},\"kind\":{\"description\":\"Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds\",\"type\":\"string\"},\"metadata\":{\"type\":\"object\"},\"status\":{\"description\":\"SecretProviderClassPodStatusStatus defines the observed state of SecretProviderClassPodStatus\",\"properties\":{\"mounted\":{\"type\":\"boolean\"},\"objects\":{\"items\":{\"description\":\"SecretProviderClassObject defines the object fetched from external secrets store\",\"properties\":{\"id\":{\"type\":\"string\"},\"version\":{\"type\":\"string\"}},\"type\":\"object\"},\"type\":\"array\"},\"podName\":{\"type\":\"string\"},\"secretProviderClassName\":{\"type\":\"string\"},\"targetPath\":{\"type\":\"string\"}},\"type\":\"object\"}},\"type\":\"object\"}},\"served\":true,\"storage\":true}]},\"status\":{\"acceptedNames\":{\"kind\":\"\",\"plural\":\"\"},\"conditions\":[],\"storedVersions\":[]}}\n"}},"spec":{"versions":[{"name":"v1alpha1","schema":{"openAPIV3Schema":{"description":"SecretProviderClassPodStatus is the Schema for the secretproviderclassespodstatus API","properties":{"apiVersion":{"description":"APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources","type":"string"},"kind":{"description":"Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds","type":"string"},"metadata":{"type":"object"},"status":{"description":"SecretProviderClassPodStatusStatus defines the observed state of SecretProviderClassPodStatus","properties":{"mounted":{"type":"boolean"},"objects":{"items":{"description":"SecretProviderClassObject defines the object fetched from external secrets store","properties":{"id":{"type":"string"},"version":{"type":"string"}},"type":"object"},"type":"array"},"podName":{"type":"string"},"secretProviderClassName":{"type":"string"},"targetPath":{"type":"string"}},"type":"object"}},"type":"object"}},"served":true,"storage":true}]},"status":{"acceptedNames":{"kind":"","plural":""},"conditions":[],"storedVersions":[]}}
to:
Resource: "apiextensions.k8s.io/v1, Resource=customresourcedefinitions", GroupVersionKind: "apiextensions.k8s.io/v1, Kind=CustomResourceDefinition"
Name: "secretproviderclasspodstatuses.secrets-store.csi.x-k8s.io", Namespace: ""
for: "crds/secrets-store.csi.x-k8s.io_secretproviderclasspodstatuses.yaml": CustomResourceDefinition.apiextensions.k8s.io "secretproviderclasspodstatuses.secrets-store.csi.x-k8s.io" is invalid: status.storedVersions[1]: Invalid value: "v1": must appear in spec.versions```
This behavior started happening when I upgraded to aks version 1.28.9.
**What did you expect to happen:**
We expected no errors. Same hem chart and configuration was working with aks version 1.27.9.
**Anything else you would like to add:**
[Miscellaneous information that will assist in solving the issue.]
**Which access mode did you use to access the Azure Key Vault instance:**
[e.g. Service Principal, Pod Identity, User Assigned Managed Identity, System Assigned Managed Identity]
**Environment:**
- Secrets Store CSI Driver version: (use the image tag): 0.2.0
- Azure Key Vault provider version: (use the image tag):
- Kubernetes version: (use `kubectl version` and `kubectl get nodes -o wide`): 1.28.9
- Cluster type: (e.g. AKS, aks-engine, etc): AKS
- Installation method: ([Helm](https://azure.github.io/secrets-store-csi-driver-provider-azure/docs/getting-started/installation/#deployment-using-helm) , [Deployment yamls](https://azure.github.io/secrets-store-csi-driver-provider-azure/docs/getting-started/installation/#using-deployment-yamls), [AKS managed add-on](https://docs.microsoft.com/en-us/azure/aks/csi-secrets-store-driver)):
Have you
What steps did you take and what happened: When I tried running below command -
helm upgrade --install --debug --wait --timeout 10m0s csi-secret-store-provider-driver csi-secret-store-provider-driver/ --namespace kube-systemI am getting error -Upgrade "csi-secret-store-provider-driver" failed: pre-upgrade hooks failed: 1 error occurred: * job secrets-store-csi-driver-upgrade-crds failed: BackoffLimitExceededWhen I saw the logs with below command -
kubectl logs job/secrets-store-csi-driver-upgrade-crds -n kube-systemI found the below error-