search

kubernetes-csi / external-provisioner

Sidecar container that watches Kubernetes PersistentVolumeClaim objects and triggers CreateVolume/DeleteVolume against a CSI endpoint
Apache License 2.0
328 stars 318 forks source link

per volume secret tokens not recognized #1148

Closed jmccormick2001 closed 2 months ago

jmccormick2001 commented 5 months ago

What happened: I'm trying to specify 'per-volume' secret templates in a StorageClass but the external-provisioner is not recognizing them for the CreateVolume 'provisioner-secret-name' secret. Instead it shows this error in the provisioner log:

Warning ProvisioningFailed 1s (x4 over 8s) infinibox-csi-driver_infinidat-csi-driver-driver-0_e943e557-81ae-4050-bb17-d212ae1faed6 failed to provision volume with StorageClass "ibox-nfs-anno-secret": error resolving value "${pvc.annotations['infinidat.com/ibox_secret']}": invalid tokens: ["pvc.annotations['infinidat.com/ibox_secret']"]

What you expected to happen: I expected the provisioner to understand the syntax of the tokens as described in this document: https://kubernetes-csi.github.io/docs/secrets-and-credentials-storage-class.html

How to reproduce it: I created a StorageClass like this: parameters:

reference secret with InfiniBox credentials

csi.storage.k8s.io/controller-expand-secret-name: ${pvc.annotations['infinidat.com/ibox_secret']}
csi.storage.k8s.io/controller-expand-secret-namespace: infinidat-csi
csi.storage.k8s.io/controller-publish-secret-name: ${pvc.annotations['infinidat.com/ibox_secret']}
csi.storage.k8s.io/controller-publish-secret-namespace: infinidat-csi
csi.storage.k8s.io/node-publish-secret-name: ${pvc.annotations['infinidat.com/ibox_secret']}
csi.storage.k8s.io/node-publish-secret-namespace: infinidat-csi
csi.storage.k8s.io/node-stage-secret-name: ${pvc.annotations['infinidat.com/ibox_secret']}
csi.storage.k8s.io/node-stage-secret-namespace: infinidat-csi
csi.storage.k8s.io/provisioner-secret-name: ${pvc.annotations['infinidat.com/ibox_secret']}
csi.storage.k8s.io/provisioner-secret-namespace: infinidat-csi

Anything else we need to know?: I guess my question is why doesn't the provisioner-secret accept the token template like the other secrets?

Environment:

hoyho commented 2 months ago

IMO, This is a historical issue. In early implementation here https://github.com/kubernetes-csi/external-provisioner/pull/69 The last parameter is a pvc pointer which is nil here,


    // Resolve provision secret credentials.
    // No PVC is provided when resolving provision/delete secret names, since the PVC may or may not exist at delete time.
    provisionerSecretRef, err := getSecretReference(provisionerSecretNameKey, provisionerSecretNamespaceKey, options.Parameters, pvName, nil)
    if err != nil {
        return nil, err
    }

that will not match any pvc.annotations token

  func getSecretReference(secretParams secretParamsMap, storageClassParams map[string]string, pvName string, pvc *v1.PersistentVolumeClaim) (*v1.SecretReference, error) {
    ...
    if pvc != nil {
            nameParams[tokenPVCNameKey] = pvc.Name
            nameParams[tokenPVCNameSpaceKey] = pvc.Namespace
            for k, v := range pvc.Annotations {
                nameParams["pvc.annotations['"+k+"']"] = v
            }
        }
    }
    ...

The reason why it does not support token ${pvc.annotations['<ANNOTATION_KEY>']} was already explained in the comment: No PVC is provided when resolving provision/delete secret names, since the PVC may or may not exist at delete time.

This caused inconsistency but still make sense, because the PVC may not exist at delete time, so it is not possible to get the annotation value from the PVC （If we reference to it).

However, With this PR got merged, the deletion secret will be stored in PV's metadata(annotations) and it can be resolved on deletion time without relying on PVC. So, I think It is possible to support templated secret in pvc annoations.