[Question] Failing at auth service credentials while injecting secrets.

[pb586b]

Note: Make sure to check out known issues (https://github.com/sparebankenvest/azure-key-vault-to-kubernetes#known-issues) before submitting

Your question I'm trying to inject secrets from Key Vault to AKS, while pod starting the injection is failing at auth service with 403 status code. Need assistance to resolve this issue as the secrets are not getting pulled into the pod. I have set policies as well.

To Reproduce Not sure how to.

Logs Pod logs: I0723 16:35:14.389830 1 version.go:31] "version info" version="" commit="a7b2d04" buildDate="2021-03-11T07:33:36Z" component="vaultenv" I0723 16:35:14.389922 1 main.go:176] "azure key vault env injector initializing" I0723 16:35:14.390189 1 main.go:245] "found original container command" cmd="/bin/sh" args=[/bin/sh -c /startService.sh msUsername msPwd] I0723 16:35:14.390236 1 authentication.go:98] "checking if current auth service credentials are stale" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/com-att-data360-dev/feedbackemail-1-5-0-d2a-7b5854499d-6xbrj?secret=akv2k8s-feedbackemail-1-5-0-d2a" I0723 16:35:14.474040 1 authentication.go:116] "failed to validate credentials" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/com-att-data360-dev/feedbackemail-1-5-0-d2a-7b5854499d-6xbrj?secret=akv2k8s-feedbackemail-1-5-0-d2a" status="403 Forbidden" statusCode=403

AKV-Injector Logs: I0723 15:30:18.293342 1 main.go:143] "found pod to mutate" pod="com-att-data360-dev/" I0723 15:30:18.293374 1 pod.go:285] "creating client certificate to use with auth service" com-att-data360-dev/="(MISSING)" I0723 15:30:18.314513 1 pod.go:292] "mutate init-containers" com-att-data360-dev/="(MISSING)" I0723 15:30:18.314539 1 pod.go:298] "mutate containers" com-att-data360-dev/="(MISSING)" I0723 15:30:18.314548 1 pod.go:116] "found container to mutate" container="com-att-data360-dev/feedbackemail" I0723 15:30:18.314556 1 pod.go:119] "checking for env vars to inject" container="com-att-data360-dev/feedbackemail" I0723 15:30:18.314578 1 pod.go:122] "found env var to inject" env="ampmd-mechid-username-secret@azurekeyvault" container="com-att-data360-dev/feedbackemail" I0723 15:30:18.314590 1 pod.go:122] "found env var to inject" env="ampmd-mechid-password-secret@azurekeyvault" container="com-att-data360-dev/feedbackemail" I0723 15:30:20.414863 1 pod.go:149] "found container arguments to use for env-injector" cmd="/bin/sh -c /startService.sh msUsername msPwd" container="com-att-data360-dev/feedbackemail" I0723 15:30:20.770593 1 pod.go:307] "containers mutated and pod updated with init-container and volumes" pod="com-att-data360-dev/"

Additional context Please provide steps to troubleshoot, I can provide further details

[kristeey]

Could you provide some additional information such as:

1. Env-injector (webhook) version.
2. How you installed the tools (e.g. helm chart) and provide the chart values set (e.g. values.yaml).
3. What kind of AKV authentication you are using (https://akv2k8s.io/security/authentication/#akv-authentication-with-the-env-injector)
4. If the cluster have any special configurations. Network policies e.t.c, service mesh.

[kristeey]

Could this be related? https://github.com/SparebankenVest/azure-key-vault-to-kubernetes/issues/162

[pb586b]

Could you provide some additional information such as:

1. Env-injector (webhook) version.
2. How you installed the tools (e.g. helm chart) and provide the chart values set (e.g. values.yaml).
3. What kind of AKV authentication you are using (https://akv2k8s.io/security/authentication/#akv-authentication-with-the-env-injector)
4. If the cluster have any special configurations. Network policies e.t.c, service mesh.

1. Env-injector version - spvest/azure-keyvault-webhook:1.2.3
2. For windows - I downloaded helm from releases: https://github.com/helm/helm/releases For Ubuntu - wget https://get.helm.sh/helm-v3.4.1-linux-amd64.tar.gz, I then unpacked it using tar We used proxy settings in deployment.yaml containers: env:
   • name: http_proxy value:
   • name: https_proxy value:
3. We are using aad-pod-identity. We have multiple namespaces for different envs within the same cluster. Accessing the key vault is done my default AKS cluster Service Principal ID and thus settings policies directly to allow connections from AKS to AKV using the same Service Principal. Helm install command helm upgrade --install akv2k8s spv-charts/akv2k8s --namespace akv2k8s --set addAzurePodIdentityException=true --set keyVaultAuth=azureCloudConfig
4. No special policies with the AKS cluster.

[kristeey]

The "failed to validate credentials" status="403 Forbidden" is a result of a failed authorization of the request at the env-injector-webhook side after the pod env injector tries to validate the auth service credentials. In other words the cred validation Get request from the app pod reaches the env-injector-webhook pod, but it fails to authorize the request. Could you install akv2k8s with global.logLevel=trace flag set? I.e.

helm upgrade --install akv2k8s spv-charts/akv2k8s --namespace akv2k8s --set addAzurePodIdentityException=true --set keyVaultAuth=azureCloudConfig --set global.logLevel=trace

This will give some additional logging in the env-injector-webhook pod that handles the request. The logs should look something like

I0730 13:14:09.429133       1 version.go:31] "version info" version="1.2.3" commit="a7b2d04" buildDate="2021-03-11T07:33:20Z" component="webhook"
I0730 13:14:09.429223       1 main.go:289] "active settings" httpPort="8080" httpPortExternal="80" tlsPort="8443" tlsPortExternal="443" mtlsPort="9443" mtlsPortExternal="9443" serveMetrics=false authType="azureCloudConfig" useAuthService=true dockerInspectionTimeout=20 cloudConfigPath="/etc/kubernetes/azure.json" logLevel="6" authServiceName="akv2k8s-envinjector" mtlsPortExternal="9443" mtlsPort="9443"
I0730 13:14:09.430754       1 main.go:358] "using cloudConfig for auth - reading credentials" file="/etc/kubernetes/azure.json"
I0730 13:14:09.431123       1 provider.go:274] "azure: using managed identity extension to retrieve access token" id="..."
I0730 13:14:09.431145       1 provider.go:281] "azure: using managed identity extension to retrieve access token" id="..."
I0730 13:14:09.431175       1 main.go:376] "checking credentials by getting authorizer"
I0730 13:14:10.353136       1 auth.go:54] "auth service ca cert" file="/var/ca-cert/tls.crt"
I0730 13:14:10.353164       1 auth.go:55] "auth service ca key" file="/var/ca-cert/tls.key"
I0730 13:14:10.353537       1 main.go:392] "serving auth validation endpoint" path=":8080/auth/{namespace}/{pod}"
I0730 13:14:10.353581       1 main.go:396] "serving health endpoint" path=":8080/healthz"
I0730 13:14:10.353807       1 main.go:443] "serving encrypted auth endpoint" path=":9443/auth"
2021/07/30 13:14:10 [WARN] no tracer active
I0730 13:14:10.354018       1 main.go:419] "serving encrypted webhook endpoint" path=":8443/pods"
I0730 13:14:10.354060       1 main.go:422] "serving encrypted healthz endpoint" path=":8443/healthz"
I0730 13:14:57.396319       1 round_trippers.go:445] GET https://10.0.0.1:443/api/v1/namespaces/akv-test 200 OK in 29 milliseconds
I0730 13:14:57.401114       1 round_trippers.go:445] GET https://10.0.0.1:443/api/v1/namespaces/akv-test/pods/secret-injection-app-77d4d5b7d8-z5c8p 200 OK in 4 milliseconds
I0730 13:14:57.413305       1 round_trippers.go:445] GET https://10.0.0.1:443/api/v1/namespaces/akv-test/pods/secret-injection-app-77d4d5b7d8-z5c8p 200 OK in 4 milliseconds
I0730 13:14:57.413922       1 clientCert.go:25] "creating x509 key pair for ca cert and key"
I0730 13:14:57.414238       1 clientCert.go:32] "parse certificate"
I0730 13:14:57.414355       1 clientCert.go:38] "generating client key"
I0730 13:14:57.445285       1 clientCert.go:44] "generating serial number"
I0730 13:14:57.445475       1 clientCert.go:66] "crating x509 certificate"
I0730 13:14:57.459476       1 round_trippers.go:445] GET https://10.0.0.1:443/api/v1/namespaces/akv-test/secrets/akv2k8s-secret-injection-app 200 OK in 11 milliseconds
I0730 13:14:57.480351       1 round_trippers.go:445] GET https://10.0.0.1:443/api/v1/namespaces/akv-test 200 OK in 7 milliseconds
I0730 13:14:57.484703       1 round_trippers.go:445] GET https://10.0.0.1:443/api/v1/namespaces/akv-test/pods/secret-injection-app-77d4d5b7d8-z5c8p 200 OK in 4 milliseconds
I0730 13:14:57.523238       1 auth.go:154] "served oauth token" pod="secret-injection-app-77d4d5b7d8-z5c8p" namespace="akv-test"

But will give 403 status code instead of 200. Also, ensure that the namespace has the azure-key-vault-env-injection: enabled label.

[pb586b]

Attached webhook logs.

logs-from-webhook-in-akv2k8s-envinjector-6bc7d8b4dd-fwms9.txt

E0802 21:06:01.496531       1 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
2021-08-02T21:06:01.496669329Z      For verbose messaging see aws.Config.CredentialsChainVerboseErrors

Don't know why it is trying to check for AWS credentials as I have never provided it anywhere.

Pod Logs:

I0802 21:12:53.843633       1 version.go:31] "version info" version="" commit="a7b2d04" buildDate="2021-03-11T07:33:36Z" component="vaultenv"
I0802 21:12:53.844410       1 main.go:176] "azure key vault env injector initializing"
I0802 21:12:53.844635       1 main.go:245] "found original container command" cmd="/bin/sh" args=[/bin/sh -c /startService.sh msUsername msPwd]
I0802 21:12:53.844683       1 authentication.go:98] "checking if current auth service credentials are stale" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/com-att-data360-dev/feedbackemail-1-5-0-d2a-7b5854499d-hrqgp?secret=akv2k8s-feedbackemail-1-5-0-d2a"
I0802 21:12:53.903064       1 authentication.go:116] "failed to validate credentials" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/com-att-data360-dev/feedbackemail-1-5-0-d2a-7b5854499d-hrqgp?secret=akv2k8s-feedbackemail-1-5-0-d2a" status="403 Forbidden" statusCode=403

Note: akv2k8s is installed in "akv2k8s" namespace while the pods are in another namespace. (If it helps)

[kristeey]

The AWS credentials error is unrelated to this case.

The webhook logs that you provided indicate that (i) the webhook is able to mutate the application container and update application pod with init-container. This is the first process that the Env-Injector does when it detects a new pod that wants secrets injected in a namespace with azure-key-vault-env-injection: enabled. After this process is done, the init-container starts and copies the azure-keyvault-env executable into /azure-keyvault/ volumepath in the application pod -> (ii) the application pod starts running the mutated container that will execute the azure-keyvault-env command. It gets theAzure Key Vault credentials by sending a GET request to the /auth/... endpoint of the webhook pod (Env-Injector auth service). This should in theory return a oauth token and everything should be good, but this is not the case here. So this means that something funky is happening in (iii) the webhook pod when it receives the GET request and tries to handle it (i.e. in the Auth Validate handler). This fails with StatusCode 403 if the authorize function returns an error, which it does if a K8S api GET requests (to get Pods or Namespace) fails, the namespace (that it gets from the K8s api) somehow does not have the azure-key-vault-env-injection label, the container (in the pod that it gets from the K8S api) does not contain the injector command, or the the pod does not have the env-injector init-container in it.

If we take a look at the timestamps of the logs that you provided, they cover the initial "mutate container" phase (i) of the webhook pod (logs-from-webhook-in-akv2k8s-envinjector-6bc7d8b4dd-fwms9.txt), and the logs of the application pods "mutated container run" phase (ii). We miss the logs from the "authentication handling" phase (iii) in the webhook pod, which I (unfortunately) would guess was handled by the second webhook pod. Hence it is hard to say exactly what happens without this last missing piece.

[pb586b]

I rescaled the webhook pods and obtaining the container logs. logs-from-webhook-in-akv2k8s-envinjector-6bc7d8b4dd-zjxbk.txt

As far as I can tell, it has the same logs. From where can I obtain further logs or how how do we troubleshoot it further?

[kristeey]

Could this be related? https://github.com/SparebankenVest/azure-key-vault-to-kubernetes/issues/175

[muwiess]

Hi @kristeey we applied all the recommendations on #175 https://github.com/SparebankenVest/azure-key-vault-to-kubernetes/issues/175 but we still see the error.

The error is happening when the injector attempts to access the Private ACR to pull the container image, however, when the injection is not applied, the container pulls the image and runs with no issues.

We tested the injection with a public image (not ACR), and the injection works following the recommendation from #175, with the Pod Identity azureidentity and azureidentityBinding but fails when pulling the image from ACR.

We follow the recommendation to add all the roles to the managed Identity but still doesn't work.

This is an example of what we run using a non-ACR image which is working.

apiVersion: v1 kind: Pod metadata: labels: app: nginx-flex-kv-podid aadpodidbinding: eastus2-amt-nprd-aks-agentpool name: nginx-flex-kv-podid namespace: com-att-amteco-dev

spec: containers:

• name: nginx-flex-kv-podid image: nginx env:

  • name: AAF_ID value: aaf-id-secret-sync@azurekeyvault volumeMounts:
  • name: test mountPath: /kvmnt readOnly: true

  volumes:

  • name: test flexVolume: driver: "azure/kv" options: usepodidentity: "true" # [OPTIONAL] if not provided, will default to "false" keyvaultname: "eastusxxxxxxxxxkv" # the name of the KeyVault keyvaultobjectnames: "aaf-id" # list of KeyVault object names (semi-colon separated) keyvaultobjecttypes: secret # list of KeyVault object types: secret, key or cert (semi-colon separated) keyvaultobjectversions: "" # [OPTIONAL] list of KeyVault object versions (semi-colon separated), will get latest if empty resourcegroup: "amt-eastus2-nprd-infr-rg" # the resource group of the KeyVault subscriptionid: "XXXXXXXXXXXXX" # the subscription ID of the KeyVault tenantid: "exxxxxxxxxxxxxxxx6" # the tenant ID of the KeyVault

$ kubectl logs nginx-flex-kv-podid -n com-att-amteco-dev I0820 16:42:56.327515 1 version.go:31] "version info" version="" commit="a375982" buildDate="2021-08-06T06:52:36Z" component="vaultenv" I0820 16:42:56.327736 1 main.go:176] "azure key vault env injector initializing" I0820 16:42:56.327972 1 main.go:245] "found original container command" cmd="/docker-entrypoint.sh" args=[/docker-entrypoint.sh nginx -g daemon off;] I0820 16:42:56.328026 1 authentication.go:98] "checking if current auth service credentials are stale" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/com-att-amteco-dev/nginx-flex-kv-podid?secret=akv2k8s-nginx-flex-kv-podid" I0820 16:42:56.388002 1 authentication.go:111] "auth service credentials ok" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/com-att-amteco-dev/nginx-flex-kv-podid?secret=akv2k8s-nginx-flex-kv-podid" I0820 16:42:56.388548 1 authentication.go:147] "requesting azure key vault oauth token" url="https://akv2k8s-envinjector.akv2k8s.svc:9443/auth/com-att-amteco-dev/nginx-flex-kv-podid" I0820 16:42:56.421024 1 authentication.go:167] "successfully received oauth token" I0820 16:42:56.595437 1 main.go:342] "secret injected into env var" azurekeyvaultsecret="com-att-amteco-dev/aaf-id-secret-sync" env="AAF_ID" I0820 16:42:56.595494 1 main.go:348] "starting process with secrets in env vars" cmd="/docker-entrypoint.sh" args=[/docker-entrypoint.sh nginx -g daemon off;] /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/ /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh /docker-entrypoint.sh: Configuration complete; ready for start up 2021/08/20 16:42:56 [notice] 1#1: using the "epoll" event method 2021/08/20 16:42:56 [notice] 1#1: nginx/1.21.1 2021/08/20 16:42:56 [notice] 1#1: built by gcc 8.3.0 (Debian 8.3.0-6) 2021/08/20 16:42:56 [notice] 1#1: OS: Linux 5.4.0-1055-azure 2021/08/20 16:42:56 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576 2021/08/20 16:42:56 [notice] 1#1: start worker processes 2021/08/20 16:42:56 [notice] 1#1: start worker process 36 2021/08/20 16:42:56 [notice] 1#1: start worker process 37 2021/08/20 16:42:56 [notice] 1#1: start worker process 38 2021/08/20 16:42:56 [notice] 1#1: start worker process 39

The error is only being displayed in the pod logs, but the injector pod doesn't show any error with trace log level.

$ kubectl logs amtadminservice-2-0-6c5ff6c645-b7nzk -n com-att-amteco-dev I0820 18:01:48.942701 1 version.go:31] "version info" version="" commit="a375982" buildDate="2021-08-06T06:52:36Z" component="vaultenv" I0820 18:01:48.942763 1 main.go:176] "azure key vault env injector initializing" I0820 18:01:48.942949 1 main.go:245] "found original container command" cmd="/bin/sh" args=[/bin/sh -c /startService.sh] I0820 18:01:48.942999 1 authentication.go:98] "checking if current auth service credentials are stale" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/com-att-amteco-dev/amtadminservice-2-0-6c5ff6c645-b7nzk?secret=akv2k8s-amtadminservice-2-0" E0820 18:01:48.993522 1 authentication.go:116] "failed to validate credentials" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/com-att-amteco-dev/amtadminservice-2-0-6c5ff6c645-b7nzk?secret=akv2k8s-amtadminservice-2-0" status="403 Forbidden" statusCode=403 I0820 18:01:48.993624 1 authentication.go:98] "checking if current auth service credentials are stale" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/com-att-amteco-dev/amtadminservice-2-0-6c5ff6c645-b7nzk?secret=akv2k8s-amtadminservice-2-0" E0820 18:01:49.008694 1 authentication.go:116] "failed to validate credentials" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/com-att-amteco-dev/amtadminservice-2-0-6c5ff6c645-b7nzk?secret=akv2k8s-amtadminservice-2-0" status="403 Forbidden" statusCode=403 I0820 18:01:52.009192 1 authentication.go:98] "checking if current auth service credentials are stale" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/com-att-amteco-dev/amtadminservice-2-0-6c5ff6c645-b7nzk?secret=akv2k8s-amtadminservice-2-0" E0820 18:01:52.063410 1 authentication.go:116] "failed to validate credentials" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/com-att-amteco-dev/amtadminservice-2-0-6c5ff6c645-b7nzk?secret=akv2k8s-amtadminservice-2-0" status="403 Forbidden" statusCode=403 I0820 18:01:58.067216 1 authentication.go:98] "checking if current auth service credentials are stale" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/com-att-amteco-dev/amtadminservice-2-0-6c5ff6c645-b7nzk?secret=akv2k8s-amtadminservice-2-0" E0820 18:01:58.078989 1 authentication.go:116] "failed to validate credentials" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/com-att-amteco-dev/amtadminservice-2-0-6c5ff6c645-b7nzk?secret=akv2k8s-amtadminservice-2-0" status="403 Forbidden" statusCode=403 E0820 18:01:58.079064 1 main.go:261] "failed to get credentials" err="failed to validate credentials, got http status code 403" failedTimes=3

kubectl logs akv2k8s-envinjector-85b6d98dc4-99nr7 -n akv2k8s I0820 17:28:14.035605 1 main.go:143] "found pod to mutate" pod="com-att-amteco-dev/" I0820 17:28:14.035631 1 pod.go:285] "creating client certificate to use with auth service" com-att-amteco-dev/="(MISSING)" I0820 17:28:14.035645 1 clientCert.go:25] "creating x509 key pair for ca cert and key" I0820 17:28:14.035836 1 clientCert.go:32] "parse certificate" I0820 17:28:14.035884 1 clientCert.go:38] "generating client key" I0820 17:28:14.056492 1 clientCert.go:44] "generating serial number" I0820 17:28:14.056518 1 clientCert.go:66] "crating x509 certificate" I0820 17:28:14.059012 1 pod.go:292] "mutate init-containers" com-att-amteco-dev/="(MISSING)" I0820 17:28:14.059030 1 pod.go:298] "mutate containers" com-att-amteco-dev/="(MISSING)" I0820 17:28:14.059056 1 pod.go:116] "found container to mutate" container="com-att-amteco-dev/amtadminservice" I0820 17:28:14.059064 1 pod.go:119] "checking for env vars to inject" container="com-att-amteco-dev/amtadminservice" I0820 17:28:14.059103 1 pod.go:122] "found env var to inject" env="aaf-id-secret-sync@azurekeyvault" container="com-att-amteco-dev/amtadminservice" I0820 17:28:14.059117 1 pod.go:122] "found env var to inject" env="aaf-password-secret-sync@azurekeyvault" container="com-att-amteco-dev/amtadminservice" I0820 17:28:14.059126 1 registry.go:30] "getting container command for container" container="com-att-amteco-dev/amtadminservice" I0820 17:28:14.059139 1 registry.go:36] "no cmd override in kubernetes for container, checking docker image configuration for entrypoint and cmd" image="amteastus2crn0rfliie.azurecr.io/com.att.amteco/amtadminservice:2.0.92" container="com-att-amteco-dev/amtadminservice" I0820 17:28:14.059171 1 registry.go:89] "found image in cache" image="amteastus2crn0rfliie.azurecr.io/com.att.amteco/amtadminservice:2.0.92" I0820 17:28:14.059187 1 pod.go:149] "found container arguments to use for env-injector" cmd="/bin/sh -c /startService.sh" container="com-att-amteco-dev/amtadminservice" I0820 17:28:14.307795 1 pod.go:265] "signed arguments to prevent override" container="com-att-amteco-dev/amtadminservice" I0820 17:28:14.307877 1 pod.go:272] "public signing key for argument verification" key="-----BEGIN RSA PUBLIC KEY-----\n\n-----END RSA PUBLIC KEY-----\n" container="com-att-amteco-dev/amtadminservice" I0820 17:28:14.307914 1 pod.go:159] "full exec path" path="/azure-keyvault/azure-keyvault-env" container="com-att-amteco-dev/amtadminservice" I0820 17:28:14.307954 1 pod.go:170] "mounting volume" volume="azure-keyvault-env" path="/azure-keyvault/" container="com-att-amteco-dev/amtadminservice" I0820 17:28:14.356924 1 round_trippers.go:454] POST https://10.0.16.1:443/api/v1/namespaces/com-att-amteco-dev/secrets 409 Conflict in 48 milliseconds I0820 17:28:14.387465 1 round_trippers.go:454] PUT https://10.0.16.1:443/api/v1/namespaces/com-att-amteco-dev/secrets/akv2k8s-amtadminservice-2-0 200 OK in 30 milliseconds I0820 17:28:14.387823 1 pod.go:307] "containers mutated and pod updated with init-container and volumes" pod="com-att-amteco-dev/" 2021/08/20 17:28:14 [DEBUG] json patch for request f080cea5-b8bb-455d-b465-5e213d7c1d15: [{"op":"add","path":"/spec/volumes/4","value":{"name":"azure-keyvault-env","emptyDir":{"medium":"Memory"}}},{"op":"add","path":"/spec/volumes/5","value":{"name":"akv2k8s-client-cert","secret":{"secretName":"akv2k8s-amtadminservice-2-0","defaultMode":420}}},{"op":"add","path":"/spec/initContainers","value":[{"name":"copy-azurekeyvault-env","image":"spvest/azure-keyvault-env:1.3.0","command":["sh","-c","cp /usr/local/bin/azure-keyvault-env /azure-keyvault/"],"resources":{},"volumeMounts":[{"name":"azure-keyvault-env","mountPath":"/azure-keyvault/"}],"imagePullPolicy":"IfNotPresent"}]},{"op":"add","path":"/spec/containers/0/command","value":["/azure-keyvault/azure-keyvault-env"]},{"op":"add","path":"/spec/containers/0/args","value":["/bin/sh","-c","/startService.sh"]},{"op":"add","path":"/spec/containers/0/env/35","value":{"name":"ENV_INJECTOR_ARGS_SIGNATURE","value":"="}},{"op":"add","path":"/spec/containers/0/env/36","value":{"name":"ENV_INJECTOR_ARGS_KEY","value":""}},{"op":"add","path":"/spec/containers/0/env/37","value":{"name":"ENV_INJECTOR_USE_AUTH_SERVICE","value":"true"}},{"op":"add","path":"/spec/containers/0/env/38","value":{"name":"ENV_INJECTOR_POD_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}}},{"op":"add","path":"/spec/containers/0/env/39","value":{"name":"ENV_INJECTOR_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}}},{"op":"add","path":"/spec/containers/0/env/40","value":{"name":"ENV_INJECTOR_CLIENT_CERT_DIR","value":"/var/client-cert/"}},{"op":"add","path":"/spec/containers/0/env/41","value":{"name":"ENV_INJECTOR_AUTH_SERVICE","value":"https://akv2k8s-envinjector.akv2k8s.svc:9443"}},{"op":"add","path":"/spec/containers/0/env/42","value":{"name":"ENV_INJECTOR_AUTH_SERVICE_VALIDATION","value":"http://akv2k8s-envinjector.akv2k8s.svc:80"}},{"op":"add","path":"/spec/containers/0/env/43","value":{"name":"ENV_INJECTOR_AUTH_SERVICE_SECRET","value":"akv2k8s-amtadminservice-2-0"}},{"op":"add","path":"/spec/containers/0/volumeMounts/4","value":{"name":"azure-keyvault-env","readOnly":true,"mountPath":"/azure-keyvault/"}},{"op":"add","path":"/spec/containers/0/volumeMounts/5","value":{"name":"akv2k8s-client-cert","readOnly":true,"mountPath":"/var/client-cert/"}}] 2021/08/20 17:28:52 [DEBUG] reviewing request e0fd0e9f-2870-49c0-9239-16481cf515c9, named: com-att-amteco-dev/

We have tried to set Debug ENV_INJECTOR_LOG_LEVEL to the container but is not working,

apiVersion: apps/v1 kind: Deployment metadata: annotations: deployment.kubernetes.io/revision: "1" meta.helm.sh/release-name: akv2k8s meta.helm.sh/release-namespace: akv2k8s creationTimestamp: "2021-08-20T16:51:09Z" generation: 2 labels: app.kubernetes.io/instance: akv2k8s app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: akv2k8s helm.sh/chart: akv2k8s-2.1.0 name: akv2k8s-envinjector namespace: akv2k8s resourceVersion: "12024461" selfLink: /apis/apps/v1/namespaces/akv2k8s/deployments/akv2k8s-envinjector uid: c5d5a36f-b688-47b1-90ff-20213ed5407e spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app.kubernetes.io/component: akv2k8s-webhook app.kubernetes.io/instance: akv2k8s app.kubernetes.io/name: akv2k8s strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: annotations: checksum/config: 0ccea9dd631ec9849a5a6d1b5888038a0bdb6b72cc47ef532dc2c08835cce9da creationTimestamp: null labels: app.kubernetes.io/component: akv2k8s-webhook app.kubernetes.io/instance: akv2k8s app.kubernetes.io/name: akv2k8s spec: containers:

• args:
  • --cloudconfig=/etc/kubernetes/azure.json
  • --version=1.3.0
  • --versionenvimage=1.3.0
  • --v=6
  • --logging-format=text env:
    • name: ENV_INJECTOR_LOG_LEVEL value: Debug
  • name: HTTP_PORT value: "8080"
  • name: HTTP_PORT_EXTERNAL value: "80"
  • name: TLS_PORT value: "8443"
  • name: TLS_PORT_EXTERNAL value: "443"
  • name: MTLS_PORT value: "9443"
  • name: MTLS_PORT_EXTERNAL value: "9443"
  • name: MTLS_PORT value: "9443"
  • name: TLS_CERT_DIR value: /var/serving-cert
  • name: CA_CERT_DIR value: /var/ca-cert
  • name: ENV_INJECTOR_EXEC_DIR value: /azure-keyvault/
  • name: WEBHOOK_AUTH_SERVICE value: akv2k8s-envinjector
  • name: AUTH_TYPE value: azureCloudConfig
  • name: USE_AUTH_SERVICE value: "true"
  • name: AZUREKEYVAULT_ENV_IMAGE value: spvest/azure-keyvault-env:1.3.0
  • name: DOCKER_IMAGE_INSPECTION_TIMEOUT value: "20"
  • name: METRICS_ENABLED value: "false" image: spvest/azure-keyvault-webhook:1.3.0 imagePullPolicy: IfNotPresent name: webhook ports:
  • containerPort: 8443 name: webhook-tls protocol: TCP
  • containerPort: 8080 name: http protocol: TCP
  • containerPort: 9443 name: auth-tls protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /healthz port: 8443 scheme: HTTPS periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 resources: {} securityContext: allowPrivilegeEscalation: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts:
  • mountPath: /var/serving-cert name: serving-cert
  • mountPath: /var/ca-cert name: ca-cert
  • mountPath: /etc/kubernetes/azure.json name: azureconf readOnly: true dnsPolicy: ClusterFirst restartPolicy: Always schedulerName: default-scheduler securityContext: {} serviceAccount: akv2k8s-envinjector serviceAccountName: akv2k8s-envinjector terminationGracePeriodSeconds: 30 volumes:
• name: serving-cert secret: defaultMode: 420 secretName: akv2k8s-envinjector-tls
• name: ca-cert secret: defaultMode: 420 secretName: akv2k8s-envinjector-ca
• hostPath: path: /etc/kubernetes/azure.json type: File name: azureconf

This is the error that we are only able to capture at the container level:

I0820 18:20:00.969580 1 version.go:31] "version info" version="" commit="a375982" buildDate="2021-08-06T06:52:36Z" component="vaultenv" I0820 18:20:00.970746 1 main.go:176] "azure key vault env injector initializing" I0820 18:20:00.970971 1 main.go:245] "found original container command" cmd="/bin/sh" args=[/bin/sh -c /startService.sh] I0820 18:20:00.971336 1 authentication.go:98] "checking if current auth service credentials are stale" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/com-att-amteco-dev/amtadminservice-2-0-6c5ff6c645-mf2sp?secret=akv2k8s-amtadminservice-2-0" E0820 18:20:00.991120 1 authentication.go:116] "failed to validate credentials" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/com-att-amteco-dev/amtadminservice-2-0-6c5ff6c645-mf2sp?secret=akv2k8s-amtadminservice-2-0" status="403 Forbidden" statusCode=403 I0820 18:20:00.991250 1 authentication.go:98] "checking if current auth service credentials are stale" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/com-att-amteco-dev/amtadminservice-2-0-6c5ff6c645-mf2sp?secret=akv2k8s-amtadminservice-2-0" E0820 18:20:01.044453 1 authentication.go:116] "failed to validate credentials" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/com-att-amteco-dev/amtadminservice-2-0-6c5ff6c645-mf2sp?secret=akv2k8s-amtadminservice-2-0" status="403 Forbidden" statusCode=403 I0820 18:20:04.046578 1 authentication.go:98] "checking if current auth service credentials are stale" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/com-att-amteco-dev/amtadminservice-2-0-6c5ff6c645-mf2sp?secret=akv2k8s-amtadminservice-2-0" E0820 18:20:04.057490 1 authentication.go:116] "failed to validate credentials" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/com-att-amteco-dev/amtadminservice-2-0-6c5ff6c645-mf2sp?secret=akv2k8s-amtadminservice-2-0" status="403 Forbidden" statusCode=403 I0820 18:20:10.061206 1 authentication.go:98] "checking if current auth service credentials are stale" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/com-att-amteco-dev/amtadminservice-2-0-6c5ff6c645-mf2sp?secret=akv2k8s-amtadminservice-2-0" E0820 18:20:10.079383 1 authentication.go:116] "failed to validate credentials" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/com-att-amteco-dev/amtadminservice-2-0-6c5ff6c645-mf2sp?secret=akv2k8s-amtadminservice-2-0" status="403 Forbidden" statusCode=403 E0820 18:20:10.079481 1 main.go:261] "failed to get credentials" err="failed to validate credentials, got http status code 403" failedTimes=3

**We have enabled the Debug log to the Azure Key Vault, and we can tell that the secrets are being pulled (first log aaf-password) by the injector, but there is also an error in the same scope time that is detected by the injector webhook and is stopping the container.

{ "time": "2021-08-17T17:47:57.1235748Z", "category": "AuditEvent", "operationName": "SecretGet", "resultType": "Success", "correlationId": "0c7c4xxxxxxxxxxxxxx28bc", "callerIpAddress": "10.0.1.75", "identity": {"claim":{"oid":"9cf4b2a4-abb7-446d-a81e-8e2728f6da45","appid":"bcfxxxxxxxxxx1"}}, "properties": {"id":"https://easxxxxkv.vault.azure.net/secrets/aaf-password/xxxxxxx","clientInfo":"Go/go1.16.5 (amd64-linux) go-autorest/v14.2.1 Azure-SDK-For-Go/v55.3.0 keyvault/2016-10-01","subnetId":"(unknown)","privateEndpointId":"/subscriptions/7a85dc6f-83db-4c57-89ee-30a4b0375ef9/resourceGroups/amt-eastus2-nprd-infr-rg/providers/Microsoft.Network/privateEndpoints/privateendpointkeyvault","httpStatusCode":200,"requestUri":"https://eaxxxxxxxxkv.vault.azure.net/secrets/**aaf-password**/?api-version=2016-10-01","isAddressAuthorized":true,"addrAuthType":"PrivateConnection","isAccessPolicyMatch":true}, "resourceId": "/SUBSCRIPTIONS/7xxxxxxxxxx0375EF9/RESOURCEGROUPS/AMT-xxxxxxxxxx-RG/PROVIDERS/MICROSOFT.KEYVAULT/VAULTS/EAXXXXXXXXXXKV", "operationVersion": "2016-10-01", "resultSignature": "OK", "durationMs": "10"}

{ "time": "2021-08-17T17:47:10.9490993Z", "category": "AuditEvent", "operationName": "KeyUnwrap", "resultType": "Success", "resultDescription": "The user, group or application 'name=Microsoft.Storage/storageaccounts;appid=26xxxxxxxca;oid=e84xxxxxx-b006-defe4ec7f410;iss=https://sts.windows.net/e741d71c-c6b6-47b0-803c-0f3b32b07556/' does not have keys unwrapKey permission on key vault 'xeastus2-amt-nprdkv;location=eastus2'. For help resolving this issue, please see https://go.microsoft.com/fwlink/?linkid=2125287", "correlationId": "b2xxxxxxxxxxx7ca", "callerIpAddress": "52.xxxxxx128", "identity": {"claim":{"oid":"e8xxxxxxxxxx410","appid":"266xxxxxxxxxxxxxxx4ca","xms_mirid":"/subscriptions/7a8xxxxxxxxx0a4b0375ef9/resourcegroups/amtxxxxxxxr-rg/providers/Microsoft.Storage/storageAccounts/amteastus2nprdst"}}, "properties": {"id":"https://eastus2-amt-nprdkv.vault.azure.net/keys/amteastus2nprdst/a8f75a4e7d0241e5b058ae503dfabec4","clientInfo":"services_xstore_xfe_HTTP_proxy/1.0","httpStatusCode":403,"requestUri":"https://easxxxxxxxxdkv.vault.azure.net/keys/amteastus2nprdst/a8xxxxxxxxxxxbec4/unwrapkey?api-version=2016-10-01","isAddressAuthorized":false,"isAccessPolicyMatch":false,"trustedService":"Microsoft.Storage/storageaccounts"}, "resourceId": "/SUBSCRIPTIONS/7A8xxxxxxxxxxxxEF9/RESOURCEGROUPS/AMT-xxxxxxxxxxxxx-RG/PROVIDERS/MICROSOFT.KEYVAULT/VAULTS/ExxxxxxxxxxKV", "operationVersion": "2016-10-01", "resultSignature": "Forbidden", "durationMs": "24"}

We tried to --set env_injector.failurePolicy=Ignore to allow the container to continue despite the error, but that option is not being configured. We reported it as bug #235

Few more things that we have tried

Using a local secrets to access the ACR: imagePullSecrets:

• name: amtlocaldockersecret

env_injector.dockerImageInspection.useAksCredentialsWithACS=false

[muwiess]

Hi, we want to update the issue @kristeey We scanned with tcpdump the pod from the aks node, and we found that the issue was related to a proxy server being pulled when the injector was trying to validate the pod local credentials via:

GET http://akv2k8s-envinjector.akv2k8s.svc:80/auth/com-att-amteco-dev/amtadminservice-2-0-6bbb475756-x5gwl?secret=akv2k8s-amtadminservice-2-0 HTTP/1.1 Host: akv2k8s-envinjector.akv2k8s.svc:80 User-Agent: Go-http-client/1.1 Accept-Encoding: gzip

And since this is an F5 proxy, it was being blocked or 403 error:

.k.8..X.}...(.P..)J3..HTTP/1.0 403 Forbidden Server: BigIP Connection: Keep-Alive Content-Length: 113

Error Code: 402 -- F5-REJECT LinkID: XXXXXX, srcIP: 10.0.1.107 blocked to akv2k8s-envinjector.akv2k8s.svc .

This is because by the time that the docker container starts and the injector pass the parameters and tries to do the validation, apparently, the proxy server is already set and tried to be used to validate the http://akv2k8s-envinjector.akv2k8s.svc:80/auth/.

Fix, example to use the non proxy:

-Dhttp.nonProxyHosts="localhost|127.0.0.1|akv2k8s-envinjector.akv2k8s.svc" \ -Dhttps.nonProxyHosts="localhost|127.0.0.1|akv2k8s-envinjector.akv2k8s.svc" \

[kristeey]

Great job finding a solution to your problem @muwiess! @pb586b Does this help in your case?

[pb586b]

Great job finding a solution to your problem @muwiess! @pb586b Does this help in your case?

Yes, I was able to remove the proxy and receive the oauth token successfully. But now I'm having issue with getting the secrets.

Pod Logs:

2021-08-25T22:08:52.937389061Z I0825 22:08:52.937191 1 authentication.go:167] "successfully received oauth token" 2021-08-25T22:08:52.984580456Z E0825 22:08:52.984475 1 main.go:313] "failed to get azurekeyvaultsecret" err="Get \"https://172.66.0.1:443/apis/spv.no/v2beta1/namespaces/com-att-data360-dev/azurekeyvaultsecrets/ampmd-aaf-password-secret\": Forbidden" azurekeyvaultsecret="com-att-data360-dev/ampmd-aaf-password-secret" 2021-08-25T22:08:52.984616056Z I0825 22:08:52.984502 1 main.go:314] "will retry getting azurekeyvaultsecret" azurekeyvaultsecret="com-att-data360-dev/ampmd-aaf-password-secret" retryTimes=3 delay=3 2021-08-25T22:09:02.108966387Z E0825 22:09:02.107603 1 main.go:326] "error getting azurekeyvaultsecret" err="Get \"https://172.66.0.1:443/apis/spv.no/v2beta1/namespaces/com-att-data360-dev/azurekeyvaultsecrets/ampmd-aaf-password-secret\": Forbidden" azurekeyvaultsecret="com-att-data360-dev/ampmd-aaf-password-secret"