search

SparebankenVest / azure-key-vault-to-kubernetes

Azure Key Vault to Kubernetes (akv2k8s for short) makes it simple and secure to use Azure Key Vault secrets, keys and certificates in Kubernetes.
https://akv2k8s.io
Apache License 2.0
439 stars 97 forks source link

Cannot schedule pod unrelated to akv2k8s - certificate signed by unknown authority #709

Open lyubomirk opened 6 months ago

lyubomirk commented 6 months ago

Note: Make sure to check out known issues (https://akv2k8s.io/troubleshooting/known-issues/) before submitting

Components and versions Select which component(s) the bug relates to with [X].

[ ] Controller, version: x.x.x (docker image tag) [x] Env-Injector (webhook), version: 1.6.0 (docker image tag) [ ] Other

Describe the bug I have an AKS cluster and a kubernetes namespace, with the label azure-key-vault-env-injection: enabled. I use env injection for only one of my workloads. The others don't reference akv2k8s at all. Still, the ones that are NOT referencing keyvault secrets sometimes fail to schedule with the following error message:

Error creating: Internal error occurred: failed calling webhook "pods.env-injector.admission.spv.no": failed to call webhook: Post "https://akv2k8s-envinjector.akv2k8s.svc:443/pods?timeout=10s": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "svc-cat-ca")

If I restart the injector pods it sometimes solves the issue temporarily, but at some point, I get the above error again.

To Reproduce Steps to reproduce the behavior:

  1. Install akv2k8s using the latest helm chart (2.6.0) and the default values file.
  2. Add label azure-key-vault-env-injection: enabled to desired namespace
  3. Schedule a pod to the namespace using a deployment (none of the workloads should reference any of the akv2k8s resources)

Expected behavior The pod should schedule without issues

Additional context I haven't seen this issue in the older versions of the helm chart (chart version 2.1.0 - Image versions 1.3.0)

lyubomirk commented 6 months ago

The issue appeared again. I receive the following error message in the envinjector: image

When I look up the IP, it is Azure's konnectiviy-agent. Restarting the pods does not help this time. There are events that the sync of the secrets is successful, but the pod that uses the secret does not get admitted.