Open dan-pulse opened 3 years ago
The err="keyvault.BaseClient#GetSecret: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded"
error is saying that the Get Secret request times out when talking to AKV. This could be due to something blocking outgoing requests from the cluster, or miss-configuration of the AzureKeyVaultSecret
object.
From experience, this error occurs if you have miss-configured the key vault name in the AzureKeyVaultSecret
object, hence the request will time out since the vault does not exist.
Ensure that the following is correct
spec:
vault:
name: <KEY VAULT NAME>
Similar issue: https://github.com/SparebankenVest/azure-key-vault-to-kubernetes/issues/127
I'm not sure that the other issue is the same as this, the weird behaviour we see for this same secret (content-repo-password):
Everything is in the same namespace so I wouldn't have thought there is anything different as both create pods it's just the one that is created via a job does not work for injecting the secret.
Could you provide the manifest for the Job
, Deployment
and AzureKeyVaultSecret
(content-repo-password
)?
akv2k8s.txt Have uploaded a text file with manifests, as a slight update as I'm just revisiting this for the first time in a while, these are the errors we now see. From the pod that needs the secret:
time="2021-07-28T12:23:46Z" level=info msg="received new Vault token" addr= app=vault-env path=kubernetes role=default
time="2021-07-28T12:23:46Z" level=info msg="initial Vault token arrived" app=vault-env
time="2021-07-28T12:23:46Z" level=info msg="renewed Vault token" app=vault-env ttl=1h0m0s
time="2021-07-28T12:23:46Z" level=info msg="spawning process: [/azure-keyvault/azure-keyvault-env /bin/sh -ex -c apt-get update && apt-get install gettext-base -y && cat /docker-entrypoint-initdb.d/provisioner.sql | envsubst | mysql -u ${MYSQL_USERNAME}]" app=vault-env
I0728 12:23:46.977665 1 version.go:31] "version info" version="" commit="92f953b" buildDate="2021-02-24T09:08:55Z" component="vaultenv"
I0728 12:23:46.977732 1 main.go:174] "azure key vault env injector initializing"
I0728 12:23:46.977894 1 main.go:224] "found original container command" cmd="/bin/sh" args=[/bin/sh -ex -c apt-get update && apt-get install gettext-base -y && cat /docker-entrypoint-initdb.d/provisioner.sql | envsubst | mysql -u ${MYSQL_USERNAME}]
I0728 12:23:46.977932 1 authentication.go:98] "checking if current auth service credentials are stale" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/q22-lab/cms-database-provisioner-ac4c5842-tnt9s?secret=akv2k8s-cms-database-provisioner"
I0728 12:23:46.992522 1 authentication.go:116] "failed to validate credentials" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/q22-lab/cms-database-provisioner-ac4c5842-tnt9s?secret=akv2k8s-cms-database-provisioner" status="403 Forbidden" statusCode=403
From the injector pod:
E0728 12:23:37.053198 1 auth.go:190] "failed to authorize request" err="no container has env-injector command" pod="cms-database-provisioner-ac4c5842-tnt9s" namespace="q22-lab"
I just did a small test in a basic AKS with kubernetes version v1.19.11 following the steps:
akv2k8s-test-keyvault
and add an access policy with the AKS service principal. Create a new secret my-test-secret
in that key vault. helm upgrade --install akv2k8s spv-charts/akv2k8s --namespace akv2k8s
AzureKeyVaultSecret
in the akv-test
namespace (that has the label azure-key-vault-env-injection: enabled
apiVersion: spv.no/v2beta1
kind: AzureKeyVaultSecret
metadata:
name: secret-sync-akv-test-namespace
namespace: akv-test
spec:
vault:
name: akv2k8s-test-keyvault # name of key vault
object:
name: my-test-secret # name of the akv secret
type: secret
Job
in the akv-test
namespace:
apiVersion: batch/v1
kind: Job
metadata:
name: secret-injection-job
namespace: akv-test
spec:
template:
spec:
containers:
- name: akv2k8s-env-test
image: spvest/akv2k8s-env-test:2.0.1 # Test image for injection
args: ["TEST_SECRET"]
env:
- name: TEST_SECRET
value: secret-sync-akv-test-namespace@azurekeyvault
restartPolicy: Never
And this works. Output of secret-injection-job
pod:
I0728 12:27:19.944877 1 version.go:31] "version info" version="" commit="a7b2d04" buildDate="2021-03-11T07:33:36Z" component="vaultenv"
I0728 12:27:19.944930 1 main.go:176] "azure key vault env injector initializing"
I0728 12:27:19.945129 1 main.go:245] "found original container command" cmd="/usr/local/bin/entrypoint.sh" args=[entrypoint.sh TEST_SECRET]
I0728 12:27:19.945173 1 authentication.go:98] "checking if current auth service credentials are stale" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/akv-test/secret-injection-job-sfjvc?secret=akv2k8s-secret-injection"
I0728 12:27:20.024202 1 authentication.go:111] "auth service credentials ok" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/akv-test/secret-injection-job-sfjvc?secret=akv2k8s-secret-injection"
I0728 12:27:20.024790 1 authentication.go:147] "requesting azure key vault oauth token" url="https://akv2k8s-envinjector.akv2k8s.svc:9443/auth/akv-test/secret-injection-job-sfjvc"
I0728 12:27:20.423365 1 authentication.go:167] "successfully received oauth token"
I0728 12:27:20.997455 1 main.go:342] "secret injected into env var" azurekeyvaultsecret="akv-test/secret-sync-akv-test-namespace" env="TEST_SECRET"
I0728 12:27:20.997523 1 main.go:348] "starting process with secrets in env vars" cmd="/usr/local/bin/entrypoint.sh" args=[entrypoint.sh TEST_SECRET]
value-of-my-test-secret
waiting forever...
yep so we see the same successful output in a pod created by a deployment:
I0728 12:41:24.572982 1 version.go:31] "version info" version="" commit="92f953b" buildDate="2021-02-24T09:08:55Z" component="vaultenv"
I0728 12:41:24.573121 1 main.go:174] "azure key vault env injector initializing"
I0728 12:41:24.573309 1 main.go:224] "found original container command" cmd="/usr/local/openjdk-11/bin/java" args=[java -cp /app/resources:/app/classes:/app/libs/* com.pulselive.cms.Conte
ntRepositoryApplication]
I0728 12:41:24.573364 1 authentication.go:98] "checking if current auth service credentials are stale" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/q22-lab/content-repo-6b9cb8ccc5-
scsmg?secret=akv2k8s-content-repo"
I0728 12:41:24.668631 1 authentication.go:111] "auth service credentials ok" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/q22-lab/content-repo-6b9cb8ccc5-scsmg?secret=akv2k8s-conte
nt-repo"
I0728 12:41:24.669145 1 authentication.go:147] "requesting azure key vault oauth token" url="https://akv2k8s-envinjector.akv2k8s.svc:9443/auth/q22-lab/content-repo-6b9cb8ccc5-scsmg"
I0728 12:41:24.810412 1 authentication.go:167] "successfully received oauth token"
I0728 12:41:25.109050 1 main.go:321] "secret injected into env var" azurekeyvaultsecret="q22-lab/content-repo-password" env="DATABASE_PASSWORD"
I0728 12:41:25.109118 1 main.go:327] "starting process with secrets in env vars" cmd="/usr/local/openjdk-11/bin/java" args=[java -cp /app/resources:/app/classes:/app/libs/* com.pulselive.
cms.ContentRepositoryApplication]
But with the pod created with the job we see the errors, I'll keep investigating to see if I can see anything else that may be going wrong.
You can also use the global.logLevel=debug
value in the helm chart for more detailed logs from the env injector.
so ran with debug logging, we get this on the env-injector pod logs:
2021/07/28 15:12:55 [DEBUG] reviewing request 0c397fba-77c8-40a2-9dbb-61271654b09f, named: q22-lab/
I0728 15:12:55.373416 1 main.go:143] "found pod to mutate" pod="q22-lab/"
I0728 15:12:55.373435 1 pod.go:285] "creating client certificate to use with auth service" q22-lab/="(MISSING)"
I0728 15:12:55.373465 1 clientCert.go:25] "creating x509 key pair for ca cert and key"
I0728 15:12:55.373607 1 clientCert.go:32] "parse certificate"
I0728 15:12:55.373675 1 clientCert.go:38] "generating client key"
I0728 15:12:55.398839 1 clientCert.go:44] "generating serial number"
I0728 15:12:55.398860 1 clientCert.go:66] "crating x509 certificate"
I0728 15:12:55.400491 1 pod.go:292] "mutate init-containers" q22-lab/="(MISSING)"
I0728 15:12:55.400503 1 pod.go:298] "mutate containers" q22-lab/="(MISSING)"
I0728 15:12:55.400511 1 pod.go:116] "found container to mutate" container="q22-lab/database-provisioner"
I0728 15:12:55.400517 1 pod.go:119] "checking for env vars to inject" container="q22-lab/database-provisioner"
I0728 15:12:55.400527 1 pod.go:122] "found env var to inject" env="mysql-password@azurekeyvault" container="q22-lab/database-provisioner"
I0728 15:12:55.400538 1 pod.go:122] "found env var to inject" env="content-repo-password@azurekeyvault" container="q22-lab/database-provisioner"
I0728 15:12:55.400550 1 pod.go:122] "found env var to inject" env="cms-api-password@azurekeyvault" container="q22-lab/database-provisioner"
I0728 15:12:55.400562 1 pod.go:122] "found env var to inject" env="user-app-password@azurekeyvault" container="q22-lab/database-provisioner"
I0728 15:12:55.400571 1 pod.go:122] "found env var to inject" env="content-metadata-password@azurekeyvault" container="q22-lab/database-provisioner"
I0728 15:12:55.400580 1 registry.go:30] "getting container command for container" container="q22-lab/database-provisioner"
I0728 15:12:55.400588 1 registry.go:51] "found cmd override in kubernetes for container, no need to inspect docker image configuration" image="q22acr.azurecr.io/registry/hub/docker/com/library/mysql:5.7" container="q22-lab/database-provisioner"
I0728 15:12:55.400609 1 pod.go:149] "found container arguments to use for env-injector" cmd="/bin/sh -ex -c apt-get update && apt-get install gettext-base -y && cat /docker-entrypoint-initdb.d/provisioner.sql | envsubst | mysql -u ${MYSQL_USERNAME}" container="q22-lab/database-provisioner"
I0728 15:12:55.553729 1 pod.go:265] "signed arguments to prevent override" container="q22-lab/database-provisioner"
I0728 15:12:55.553795 1 pod.go:272] "public signing key for argument verification" key="-----BEGIN RSA PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAv8Q7PXPiPrOsr9ox1maB\njuWPxl0/P+GLyHslE9gWtFO4BirhrGlOdIQWj27WrjxkNR6CDXb94ea6y/i1gbD7\n2EJg01YMs5odFbyG01F2JWBFhEDmvmf2g70EcRJ2ppPIk3aG+Njeo2sesqvz/v0q\nAWoNh9T+ovd9gB6uvVwDsrfaock1cAgO8j5pOdUAsBpsEszbibZxSf0sBK8lG7Ku\nBly2MLrFAlatqWJykdERsN5XUdAJGPs5XnG5o28aSzicxRi/6mG9x+4SYn3/RsFe\nEebr8nUaguJssTNWsv/yGaETMlaR4fR3Jr7zA3RDpCbpq4T5QFRfoYQTeHpsSzSv\n7wIDAQAB\n-----END RSA PUBLIC KEY-----\n" container="q22-lab/database-provisioner"
I0728 15:12:55.553811 1 pod.go:159] "full exec path" path="/azure-keyvault/azure-keyvault-env" container="q22-lab/database-provisioner"
I0728 15:12:55.553822 1 pod.go:170] "mounting volume" volume="azure-keyvault-env" path="/azure-keyvault/" container="q22-lab/database-provisioner"
I0728 15:12:55.602203 1 pod.go:307] "containers mutated and pod updated with init-container and volumes" pod="q22-lab/"
2021/07/28 15:12:55 [DEBUG] json patch for request 0c397fba-77c8-40a2-9dbb-61271654b09f: [{"op":"add","path":"/spec/volumes/2","value":{"name":"azure-keyvault-env","emptyDir":{"medium":"Memory"}}},{"op":"add","path":"/spec/volumes/3","value":{"name":"akv2k8s-client-cert","secret":{"secretName":"akv2k8s-cms-database-provisioner","defaultMode":420}}},{"op":"add","path":"/spec/initContainers","value":[{"name":"copy-azurekeyvault-env","image":"spvest/azure-keyvault-env:1.2.0","command":["sh","-c","cp /usr/local/bin/azure-keyvault-env /azure-keyvault/"],"resources":{},"volumeMounts":[{"name":"azure-keyvault-env","mountPath":"/azure-keyvault/"}],"imagePullPolicy":"IfNotPresent"}]},{"op":"remove","path":"/spec/containers/0/command/2"},{"op":"remove","path":"/spec/containers/0/command/1"},{"op":"replace","path":"/spec/containers/0/command/0","value":"/azure-keyvault/azure-keyvault-env"},{"op":"add","path":"/spec/containers/0/args/0","value":"-c"},{"op":"add","path":"/spec/containers/0/args/0","value":"-ex"},{"op":"add","path":"/spec/containers/0/args/0","value":"/bin/sh"},{"op":"add","path":"/spec/containers/0/env/11","value":{"name":"ENV_INJECTOR_ARGS_SIGNATURE","value":"S1NZK3ZseFNJQS90aFZSbmhhd1RZd3F0SzBjaWJNMUJOQStnbVRUOUIvTlFqSXRreVp6a0pPTFd3cnl1Z2orRGppQjF2Kzh6K2d5VUR2Z0E2T1pKNVJKc1d2VEFQczh4OVl4THMzMkdyQVNGM1lNU2plMUt1Rm5Gck5yY3AvUjc0d2tBTmtqa1QrTU5jOGRlb0lXcXVpaFExakhxMXdUQ3A4RGF4Y2F0UG5XdkZselBFUnVWTDNPTzhpVFo0UnJhMlp3OFRNdVlwU3RYSHdvSko4eEROaGtkWkI5bUJkeVBBNlI2OFErZjMzQTZTdVNUbmIxZFZDZHlMV1ZHc21MaGpJTEdSWTRJUnVOY1NrRnUyalNuS1d3bExVZ21RbVNYMWhuUTAxVlkxdzdHUlhMMnBHcW1qSktFR0d2TzBUYnFoTTN2U050ZW8xU2VDSE1FZisyUUpBPT0="}},{"op":"add","path":"/spec/containers/0/env/12","value":{"name":"ENV_INJECTOR_ARGS_KEY","value":"LS0tLS1CRUdJTiBSU0EgUFVCTElDIEtFWS0tLS0tCk1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBdjhRN1BYUGlQck9zcjlveDFtYUIKanVXUHhsMC9QK0dMeUhzbEU5Z1d0Rk80QmlyaHJHbE9kSVFXajI3V3JqeGtOUjZDRFhiOTRlYTZ5L2kxZ2JENwoyRUpnMDFZTXM1b2RGYnlHMDFGMkpXQkZoRURtdm1mMmc3MEVjUkoycHBQSWszYUcrTmplbzJzZXNxdnovdjBxCkFXb05oOVQrb3ZkOWdCNnV2VndEc3JmYW9jazFjQWdPOGo1cE9kVUFzQnBzRXN6YmliWnhTZjBzQks4bEc3S3UKQmx5Mk1MckZBbGF0cVdKeWtkRVJzTjVYVWRBSkdQczVYbkc1bzI4YVN6aWN4UmkvNm1HOXgrNFNZbjMvUnNGZQpFZWJyOG5VYWd1SnNzVE5Xc3YveUdhRVRNbGFSNGZSM0pyN3pBM1JEcENicHE0VDVRRlJmb1lRVGVIcHNTelN2Cjd3SURBUUFCCi0tLS0tRU5EIFJTQSBQVUJMSUMgS0VZLS0tLS0K"}},{"op":"add","path":"/spec/containers/0/env/13","value":{"name":"ENV_INJECTOR_USE_AUTH_SERVICE","value":"true"}},{"op":"add","path":"/spec/containers/0/env/14","value":{"name":"ENV_INJECTOR_CLIENT_CERT_DIR","value":"/var/client-cert/"}},{"op":"add","path":"/spec/containers/0/env/15","value":{"name":"ENV_INJECTOR_POD_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}}},{"op":"add","path":"/spec/containers/0/env/16","value":{"name":"ENV_INJECTOR_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}}},{"op":"add","path":"/spec/containers/0/env/17","value":{"name":"ENV_INJECTOR_AUTH_SERVICE","value":"https://akv2k8s-envinjector.akv2k8s.svc:9443"}},{"op":"add","path":"/spec/containers/0/env/18","value":{"name":"ENV_INJECTOR_AUTH_SERVICE_VALIDATION","value":"http://akv2k8s-envinjector.akv2k8s.svc:80"}},{"op":"add","path":"/spec/containers/0/env/19","value":{"name":"ENV_INJECTOR_AUTH_SERVICE_SECRET","value":"akv2k8s-cms-database-provisioner"}},{"op":"add","path":"/spec/containers/0/volumeMounts/2","value":{"name":"azure-keyvault-env","readOnly":true,"mountPath":"/azure-keyvault/"}},{"op":"add","path":"/spec/containers/0/volumeMounts/3","value":{"name":"akv2k8s-client-cert","readOnly":true,"mountPath":"/var/client-cert/"}}]
I0728 15:12:57.199618 1 clientCert.go:25] "creating x509 key pair for ca cert and key"
I0728 15:12:57.199762 1 clientCert.go:32] "parse certificate"
I0728 15:12:57.199822 1 clientCert.go:38] "generating client key"
I0728 15:12:57.210168 1 clientCert.go:44] "generating serial number"
I0728 15:12:57.210185 1 clientCert.go:66] "crating x509 certificate"
I0728 15:12:57.244995 1 auth.go:154] "served oauth token" pod="cms-database-provisioner-ac4c5842-72mqm" namespace="q22-lab"
And then the logs for the pod that fails to get the secet:
I0728 15:13:28.224939 1 version.go:31] "version info" version="" commit="92f953b" buildDate="2021-02-24T09:08:55Z" component="vaultenv"
I0728 15:13:28.225056 1 main.go:174] "azure key vault env injector initializing"
I0728 15:13:28.225190 1 main.go:224] "found original container command" cmd="/bin/sh" args=[/bin/sh -ex -c apt-get update && apt-get install gettext-base -y && cat /docker-entrypoint-initdb.d/provisioner.sql | envsubst | mysql -u ${MYSQL_USERNAME}]
I0728 15:13:28.225227 1 authentication.go:98] "checking if current auth service credentials are stale" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/q22-lab/cms-database-provisioner-ac4c5842-72mqm?secret=akv2k8s-cms-database-provisioner"
I0728 15:13:28.292589 1 authentication.go:111] "auth service credentials ok" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/q22-lab/cms-database-provisioner-ac4c5842-72mqm?secret=akv2k8s-cms-database-provisioner"
I0728 15:13:28.292897 1 authentication.go:147] "requesting azure key vault oauth token" url="https://akv2k8s-envinjector.akv2k8s.svc:9443/auth/q22-lab/cms-database-provisioner-ac4c5842-72mqm"
I0728 15:13:28.321809 1 authentication.go:167] "successfully received oauth token"
E0728 15:13:58.346957 1 main.go:313] "failed to read secret from azure key vault" err="keyvault.BaseClient#GetSecret: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded" azurekeyvaultsecret="q22-lab/content-repo-password"
As a note it seems to fail to get a different azurekeyvaultsecret each time as there are multiple but I don't think this is related to the issue.
Any update on this? I'm struggling with an issue that looks very much like this. The problem appears only on some pods, all pods them are managed through ArgoCD.
I just realized: the only pods that fail are those with init containers. The interesting thing is that I had tried using init containers with an example deployment and that definitely works. Here's the log of the injector:
I0916 15:10:46.580501 1 registry.go:30] "getting container command for container" container="forecast-customer-str/keycloak"
I0916 15:10:46.580691 1 registry.go:36] "no cmd override in kubernetes for container, checking docker image configuration for entrypoint and cmd" image="<redacted>" container="forecast-customer-str/keycloak"
I0916 15:10:46.580728 1 registry.go:89] "found image in cache" image="<redacted>"
I0916 15:10:46.580747 1 pod.go:149] "found container arguments to use for env-injector" cmd="<redacted>" container="forecast-customer-str/keycloak"
I0916 15:10:46.594597 1 acr.go:167] "adding ACR docker config entry" url="<redacted>"
I0916 15:10:46.594761 1 acr.go:85] "found acr gredentials" url="<redacted>"
I0916 15:10:46.594927 1 acr.go:73] using managed identity for acr credentials
I0916 15:10:46.595072 1 provider.go:274] "azure: using managed identity extension to retrieve access token" id="cfb9ca4e-4b0f-402e-b121-5195ccbf751a"
I0916 15:10:46.595168 1 provider.go:281] "azure: using managed identity extension to retrieve access token" id="cfb9ca4e-4b0f-402e-b121-5195ccbf751a"
I0916 15:10:46.607838 1 acr.go:155] "discovering auth redirects" url="<redacted>"
I0916 15:10:46.609376 1 acr.go:161] exchanging an acr refresh_token
I0916 15:10:46.618957 1 acr.go:167] "adding ACR docker config entry" url="<redacted>"
I0916 15:10:46.619106 1 acr.go:85] "found acr gredentials" url="<redacted>"
I0916 15:10:46.619287 1 acr.go:73] using managed identity for acr credentials
I0916 15:10:46.619435 1 provider.go:274] "azure: using managed identity extension to retrieve access token" id="cfb9ca4e-4b0f-402e-b121-5195ccbf751a"
I0916 15:10:46.619530 1 provider.go:281] "azure: using managed identity extension to retrieve access token" id="cfb9ca4e-4b0f-402e-b121-5195ccbf751a"
I0916 15:10:46.630360 1 acr.go:155] "discovering auth redirects" url="<redacted>"
I0916 15:10:46.631934 1 acr.go:161] exchanging an acr refresh_token
I0916 15:10:46.688449 1 acr.go:167] "adding ACR docker config entry" url="<redacted>"
I0916 15:10:46.688705 1 acr.go:85] "found acr gredentials" url="<redacted>"
I0916 15:10:46.690011 1 acr.go:167] "adding ACR docker config entry" url="<redacted>"
I0916 15:10:46.690169 1 acr.go:85] "found acr gredentials" url="<redacted>"
I0916 15:10:46.768506 1 pod.go:265] "signed arguments to prevent override" container="forecast-customer-str/keycloak"
I0916 15:10:46.768594 1 pod.go:272] "public signing key for argument verification" key="-----BEGIN RSA PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA6hjv8Ty0eNtl5CniWU+/\nl/4dycJEHjNbSxLdg8nnVN+IgTE8oLcrj3H4+57ZCbuEAzs8dEZsq3Td5Hof13tP\nPPDj6rjaj4H9MsFdXbVu2ZmyO1W3IqrHcKTO8cF9n/TMEN6FveTErmtzUQIAcroP\n4gtIdi0isit2Ku1Hd31mXGCP7KpPyS6951xW3gknfQ9H7in0CJiw3G+I22GS4f+F\nnjy7fWbN1yWh7bkZU3zxQm+HbH8GG81dnw8aYC8ahbdxLbG/KzpNA8TQ2Ps8Zww2\n0MI460KirfqM+FFZ0nvHRN/R2EIH3okiuA/GRj9g2A4THsGS8R9qqUxIBKMFoygf\ngQIDAQAB\n-----END RSA PUBLIC KEY-----\n" container="forecast-customer-str/keycloak"
I0916 15:10:46.768618 1 pod.go:159] "full exec path" path="/azure-keyvault/azure-keyvault-env" container="forecast-customer-str/keycloak"
I0916 15:10:46.768641 1 pod.go:170] "mounting volume" volume="azure-keyvault-env" path="/azure-keyvault/" container="forecast-customer-str/keycloak"
I0916 15:10:46.777980 1 round_trippers.go:454] POST https://10.0.0.1:443/api/v1/namespaces/forecast-customer-str/secrets 409 Conflict in 8 milliseconds
I0916 15:10:46.782746 1 round_trippers.go:454] PUT https://10.0.0.1:443/api/v1/namespaces/forecast-customer-str/secrets/akv2k8s-keycloak 200 OK in 4 milliseconds
I0916 15:10:46.783273 1 pod.go:307] "containers mutated and pod updated with init-container and volumes" pod="forecast-customer-str/"
2
My issue is probably unrelated. I've opened a separate issue: #254
Please see my comment here: bug #127.
TL;DR
There are no certificates installed so you may run an init container first (kalaksi/ca-certificates) to populate /etc/ssl/certs
. Then you attach that directory to your container and you should be good.
Components and versions Select which component(s) the bug relates to with [X].
[ ] Controller, version:
x.x.x
(docker image tag) [ x] Env-Injector (webhook), version:1.2.0
(docker image tag) [ ] OtherDescribe the bug When deploying a job the env secrets do not get injected into the environment, this works fine when we use helm to dpeloy our services such as deployments/replica sets.
To Reproduce 1.19 AKS cluster deploy a pod with secrets from azure key vualt
Expected behavior Secrets injected into the pod created by the job
Logs If applicable, add logs to help explain your problem.
Env settings in pod created from the job:
Azurekeyvaultsecret exists:
Additional context The same secrets are pulled into a pod created from a deployment.