search

Azure / secrets-store-csi-driver-provider-azure

Azure Key Vault provider for Secret Store CSI driver allows you to get secret contents stored in Azure Key Vault instance and use the Secret Store CSI driver interface to mount them into Kubernetes pods.
https://azure.github.io/secrets-store-csi-driver-provider-azure/
MIT License
436 stars 194 forks source link

secrets-store-csi-driver is not working in Azure Stack Hub - AKS 1.20.11 #1000

Closed zohebs341 closed 1 year ago

zohebs341 commented 1 year ago

Hi Team,

By using aks-engine I've deployed the AKS cluster in Azure Stack Hub. As we have Vault in Azure Stack Hub, I want to leverage the secret-store-csi-driver plus my application needs this secrets-store-csi-driver. I've deployed secret-store-csi-drive in azure stack hub, pods are up/running. However, app/pod is not consuming it and it's failing with the below error.

Does secret-store-csi-driver works with azure aks 1.20.11?

The chart is used for deployment in Azure Stack Hub.

helm repo add csi-secrets-store-provider-azure https://azure.github.io/secrets-store-csi-driver-provider-azure/charts helm install csi csi-secrets-store-provider-azure/csi-secrets-store-provider-azure --namespace kube-system

Error Logs:

Warning FailedMount pod/api-77d6584cb-4bwg8 Unable to attach or mount volumes: unmounted volumes=[secret], unattached volumes=[secret default-token-nb7x7]: timed out waiting for the condition 17m Warning FailedMount pod/api-77d6584cb-4bwg8 Unable to attach or mount volumes: unmounted volumes=[secret], unattached volumes=[default-token-nb7x7 secret]: timed out waiting for the condition 60m Normal Scheduled pod/azure-read-api-77d6584cb-54rrl Successfully assigned api-77d6584cb-54rrl to k8s-api1-63284788-1 20m Warning FailedMount pod/api-77d6584cb-54rrl MountVolume.SetUp failed for volume "secret" : rpc error: code = DeadlineExceeded desc = context deadline exceeded

aramase commented 1 year ago

Here is our troubleshooting guide: https://azure.github.io/secrets-store-csi-driver-provider-azure/docs/troubleshooting/ to get logs from the provider. If you're using AzureStackCloud, you need to configure the keyvault, AAD endpoint and mount the custom file (ref: https://azure.github.io/secrets-store-csi-driver-provider-azure/docs/configurations/custom-environments/)

zohebs341 commented 1 year ago

@aramase Thanks for your response. As I am a new user of Azure Stack Hub, will you be able to share more details or steps for this customization?

By using the below command, I've got the below YAML file.

helm show values csi-secrets-store-provider-azure/csi-secrets-store-provider-azure > csi-azstack-values.yaml

cat csi-azstack-values.yaml

nameOverride: "" fullnameOverride: ""

One or more secrets to be used when pulling images

imagePullSecrets: []

- name: myRegistryKeySecretName

logging format json

Default is text.

logFormatJSON: false

log level. Uses V logs (klog)

logVerbosity: 0

enableArcExtension: false

linux: image: repository: mcr.microsoft.com/oss/azure/secrets-store/provider-azure tag: v1.3.0 pullPolicy: IfNotPresent nodeSelector: {} tolerations: [] enabled: true resources: requests: cpu: 50m memory: 100Mi limits: cpu: 50m memory: 100Mi podLabels: {} podAnnotations: {} priorityClassName: "" updateStrategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 privileged: false

If provided, the userAgent string will be appended to the

AKV provider user agents for all adal and keyvault requests.

customUserAgent: "" healthzPort: 8989 healthzPath: "/healthz" healthzTimeout: "5s" volumes:

windows: image: repository: mcr.microsoft.com/oss/azure/secrets-store/provider-azure tag: v1.3.0 pullPolicy: IfNotPresent nodeSelector: {} tolerations: [] enabled: false resources: requests: cpu: 100m memory: 200Mi limits: cpu: 100m memory: 200Mi podLabels: {} podAnnotations: {} priorityClassName: "" updateStrategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1

If provided, the userAgent string will be appended to the

AKV provider user agents for all adal and keyvault requests.

customUserAgent: "" healthzPort: 8989 healthzPath: "/healthz" healthzTimeout: "5s" volumes: [] volumeMounts: [] kubeletRootDir: C:\var\lib\kubelet providersDir: C:\k\secrets-store-csi-providers affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms:

Configuration values for the secrets-store-csi-driver dependency.

ref: https://github.com/kubernetes-sigs/secrets-store-csi-driver/tree/master/charts/secrets-store-csi-driver/README.md

secrets-store-csi-driver: install: true

By default helm will append the chart release name to the dependent chart names.

Explicitly setting the fullnameOverride will override this behavior.

fullnameOverride: secrets-store-csi-driver linux: enabled: true kubeletRootDir: /var/lib/kubelet metricsAddr: ":8080" priorityClassName: "" image: repository: mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver tag: v1.2.3 pullPolicy: IfNotPresent registrarImage: repository: mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar tag: v2.5.1 pullPolicy: IfNotPresent livenessProbeImage: repository: mcr.microsoft.com/oss/kubernetes-csi/livenessprobe tag: v2.7.0 pullPolicy: IfNotPresent crds: image: repository: mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver-crds tag: v1.2.3 pullPolicy: IfNotPresent providersDir: /var/run/secrets-store-csi-providers

windows: enabled: false kubeletRootDir: C:\var\lib\kubelet metricsAddr: ":8080" priorityClassName: "" image: repository: mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver tag: v1.2.3 pullPolicy: IfNotPresent registrarImage: repository: mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar tag: v2.5.1 pullPolicy: IfNotPresent livenessProbeImage: repository: mcr.microsoft.com/oss/kubernetes-csi/livenessprobe tag: v2.7.0 pullPolicy: IfNotPresent

enableSecretRotation: false rotationPollInterval: 2m

Refer to https://secrets-store-csi-driver.sigs.k8s.io/load-tests.html for more details on actions to take before enabling this feature

filteredWatchSecret: true

syncSecret: enabled: false

tokenRequests:

Install default service account

rbac: install: true pspEnabled: false

explicitly reconstruct the pem chain in the order: SERVER, INTERMEDIATE, ROOT

constructPEMChain: true

zohebs341 commented 1 year ago

@aramase @jadarsie Can you please help me with installation steps or yaml file for secret-store-csi deployment in Azure Stack Hub.

jadarsie commented 1 year ago

Please let us know what's wrong with the existing documentation. Did you try this? https://azure.github.io/secrets-store-csi-driver-provider-azure/docs/configurations/custom-environments/#update-secret-provider-class

jadarsie commented 1 year ago

In your case, it would be

cloudName: "AzureStackCloud"
cloudEnvFileName: "/etc/kubernetes/azurestackcloud.json"
zohebs341 commented 1 year ago

helm repo add csi-secrets-store-provider-azure https://azure.github.io/secrets-store-csi-driver-provider-azure/charts helm install csi csi-secrets-store-provider-azure/csi-secrets-store-provider-azure --namespace kube-system ---> Instead of installing with this command. I've created a file by using this --> helm show values csi-secrets-store-provider-azure/csi-secrets-store-provider-azure > csi-azstack-values.yaml

In that csi-azstack-values.yaml----> I believe volumes section is same for me, how about the volumeMounts

linux: volumes:

Then I need to deploy using - helm install csi -f csi-azstack-values.yaml csi-secrets-store-provider-azure/csi-secrets-store-provider-azure --namespace kube-system

is it correct? Once I deploy any app(abc), then I will get "SecretProviderClass" as a part of deployment. In that "SecretProviderClass", I've to add below paramaters and then restart the respective deployment.

cloudName: "AzureStackCloud" cloudEnvFileName: "/etc/kubernetes/azurestackcloud.json"

jadarsie commented 1 year ago

I think you got it, I would use type instead of subPath for security reasons. Maybe the ssl dir is not needed either (not sure).

linux:
  volumes:
  - name: custom-environment
    hostPath:
      path: /etc/kubernetes/azurestackcloud.json
      type: FileOrCreate
  volumeMounts:
  - name: custom-environment
    mountPath: /etc/kubernetes/azurestackcloud.json
    readOnly: true

Also, you will likely need this.

zohebs341 commented 1 year ago

@jadarsie Thanks Javier. I'm testing it and will update you ASAP.

service principal already I've deployed and taken care.

Thank you once again for your time.

zohebs341 commented 1 year ago

@jadarsie I've tested it with one basic deployment and it worked, please don't close the issue as I will deploy a few more apps in the next two days. Thank you once again for your support.

@aramase Thank you, Anish.

It is not related to you, but just a generic question/feedback as a user: I am not supposed to say this, but logically speaking: These two drivers (secrets-store-csi-drivers & azuredisk-csi-driver) should be a part of Microsoft Azure Stack Hub support policies. The reason why I am saying this is, as a user/customer: we are using/paying Azure Key Vault & Azure Storage in Azure Stack Hub. In order to use it or avail of it, above mentioned drivers are mandatory. So at least Azure Stack Hub tech support should be aware of this issue & solution. Currently, it's like you just pay for Azure Key Vault & Azure Storage, and for drivers, bring your own drivers or check it with driver owners like you.

aramase commented 1 year ago

I am not supposed to say this, but logically speaking: These two drivers (secrets-store-csi-drivers & azuredisk-csi-driver) should be a part of Microsoft Azure Stack Hub support policies. The reason why I am saying this is, as a user/customer: we are using/paying Azure Key Vault & Azure Storage in Azure Stack Hub. In order to use it or avail of it, above mentioned drivers are mandatory. So at least Azure Stack Hub tech support should be aware of this issue & solution. Currently, it's like you just pay for Azure Key Vault & Azure Storage, and for drivers, bring your own drivers or check it with driver owners like you.

@zohebs341 Thank you for the feedback. From the driver project perspective, I can say if you're using the driver and provider from OSS then support is via GitHub ([Support policy](I https://github.com/Azure/secrets-store-csi-driver-provider-azure#support)). If you're on AKS, you can use the AKS managed add-on and it's part of AKS support policy. Similarly for Azure Stack Hub, I would recommend reaching out to the ASH tech support to see if they can support this/this is integrated as part of the solution. Hope this helps!

please don't close the issue as I will deploy a few more apps in the next two days

Please let us know when you've deployed the app and this can be closed.

zohebs341 commented 1 year ago

@aramase thank you. For sure I will update you asap. Regarding support, I've already check with ASH. They are saying it's not under support policy. Both secret-store & azuredisk-csi driver's. That's why i thought to share my views here.

aramase commented 1 year ago

@zohebs341 Any updates on the issue?

zohebs341 commented 1 year ago

@aramase @jadarsie Thanks a lot. It worked for me.

I really appreciate your support on this.

aramase commented 1 year ago

That's great! @jadarsie has added documentation for this with https://github.com/Azure/secrets-store-csi-driver-provider-azure/pull/1010. I'll close this issue now. Please feel free to reopen if you have any questions.