tspearconquest
commented
1 year ago Hello!
I can see your kubelet Managed Identity is 5222b31a-2198-4fb6-9c0b-90c9a42d5d14, and that AKV2K8S is properly using this identity, and that az aks check-acr output also confirms this identity has pull permissions.
These lines below are the last 2 lines before your failure. The last line indicates that the pod doesn't have a command block. This isn't strictly required, however if you were to add one, it should work around the issue for you. What's happening is that akv2k8s is looking for the command block in order to override it to inject the azure keyvault binary as the first command in the container, so that the entrypoint of the container starts with the environment variables already populated with the values of the secrets in Keyvault. When it doesn't find that block in your pod, it has to download the image from ACR for itself in order to inspect it to find out the correct entrypoint so that it can add it after the mutation of the pod is completed.
I0228 00:58:31.336985 1 registry.go:30] "getting container command for container" container="hgphs/attphs-rasp-image"
I0228 00:58:31.336999 1 registry.go:36] "no cmd override in kubernetes for container, checking docker image configuration for entrypoint and cmd" image="28363devopsacr.azurecr.io/attphs-rasp:4.3.4.557" container="hgphs/attphs-rasp-image"As to why this is failing when the identity has permissions... I'm not certain. It may help to know how you installed akv2k8s env-injector. Did you use helm? If so, what does your values.yaml look like, and what --set flags are you passing to the helm install or helm upgrade command?
MDXdarianhuotari
abhilashjoseph
Note: Make sure to check out known issues (https://github.com/sparebankenvest/azure-key-vault-to-kubernetes#known-issues) before submitting
Your question
How do I verify default aks credentials are being used, I believe i've supplied the aks cluster default credentials (i think??) Reader/acrPull to the ACR but It looks like i've missed some authorization somewhere. Or is this issue not an authorization issue and related to the webhook? I'm lost and looking for help here!! I'm not sure if this is the seat to keyboard interface thats the problem, or if i am actually running into a bug?
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION akv2k8s akv2k8s 1 2023-02-27 18:41:31.893585041 +0000 UTC deployed akv2k8s-2.3.0 1.4.0
Error creating: Internal error occurred: failed calling webhook "pods.env-injector.admission.spv.no": failed to call webhook: an error on the server ("{\"response\":{\"uid\":\"b05d078f-8bff-4ca7-a3fc-78a445697016\",\"allowed\":false,\"status\":{\"metadata\":{},\"status\":\"Failure\",\"message\":\"failed to get auto cmd, error: GET https://28363devopsacr.azurecr.io/oauth2/token?scope=repository%3Aattphs-rasp%3Apull\\u0026service=28363devopsacr.azurecr.io: UNAUTHORIZED: authentication required, visit https://aka.ms/acr/authorization for more information.\ncannot fetch image descriptor\ngithub.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry.getImageConfig\n\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry/registry.go:144\ngithub.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry.(Registry).GetImageConfig\n\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry/registry.go:103\nmain.getContainerCmd\n\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/registry.go:39\nmain.podWebHook.mutateContainers\n\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/pod.go:143\nmain.podWebHook.mutatePodSpec\n\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/pod.go:299\nmain.vaultSecretsMutator\n\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/main.go:163\ngithub.com/slok/kubewebhook/pkg/webhook/mutating.MutatorFunc.Mutate\n\t/go/pkg/mod/github.com/slok/kubewebhook@v0.11.0/pkg/webhook/mutating/mutator.go:25\ngithub.com/slok/kubewebhook/pkg/webhook/mutating.mutationWebhook.mutatingAdmissionReview\n\t/go/pkg/mod/github.com/slok/kubewebhook@v0.11.0/pkg/webhook/mutating/webhook.go:128\ngithub.com/slok/kubewebhook/pkg/webhook/mutating.mutationWebhook.Review\n\t/go/pkg/mod/github.com/slok/kubewebhook@v0.11.0/pkg/webhook/mutating/webhook.go:120\ngithub.com/slok/kubewebhook/pkg/webhook/internal/instrumenting.(Webhook).Review\n\t/go/pkg/mod/github.com/slok/kubewebhook@v0.11.0/pkg/webhook/internal/ins") has prevented the request from succeeding
To Reproduce If question relates to a certain behavior, describe steps to reproduce:
Logs If applicable, add logs to help add context to your question.
$ az aks check-acr --name hgphs-westus2-dr-aks-01 --resource-group hgphs-westus2-dr-rg-01 --acr 28363devopsacr.azurecr.io Merged "hgphs-westus2-dr-aks-01" as current context in /tmp/tmpriic11sh WARNING: version difference between client (1.26) and server (1.24) exceeds the supported minor version skew of +/-1
[2023-02-28T01:34:47Z] Checking host name resolution (28363devopsacr.azurecr.io): SUCCEEDED [2023-02-28T01:34:47Z] Canonical name for ACR (28363devopsacr.azurecr.io): 28363devopsacr.privatelink.azurecr.io. [2023-02-28T01:34:47Z] Checking managed identity... [2023-02-28T01:34:47Z] Kubelet managed identity client ID: 5222b31a-2198-4fb6-9c0b-90c9a42d5d14 [2023-02-28T01:34:48Z] Validating managed identity existance: SUCCEEDED [2023-02-28T01:34:48Z] Validating image pull permission: SUCCEEDED [2023-02-28T01:34:48Z] Your cluster can pull images from 28363devopsacr.azurecr.io!
$ k logs akv2k8s-envinjector-54f6b56cc9-9xh5r -n akv2k8s I0227 18:41:34.179198 1 version.go:31] "version info" version="1.4.0" commit="15d87b2" buildDate="2022-12-08T21:19:10Z" component="webhook" I0227 18:41:34.179282 1 main.go:290] "active settings" httpPort="8080" httpPortExternal="80" tlsPort="8443" tlsPortExternal="443" mtlsPort="9443" mtlsPortExternal="9443" serveMetrics=false authType="azureCloudConfig" useAuthService=true dockerInspectionTimeout=20 cloudConfigPath="/etc/kubernetes/azure.json" logLevel="4" authServiceName="akv2k8s-envinjector" mtlsPortExternal="9443" mtlsPort="9443" I0227 18:41:34.179581 1 main.go:359] "using cloudConfig for auth - reading credentials" file="/etc/kubernetes/azure.json" I0227 18:41:34.179910 1 provider.go:274] "azure: using managed identity extension to retrieve access token" id="5222b31a-2198-4fb6-9c0b-90c9a42d5d14" I0227 18:41:34.179926 1 provider.go:281] "azure: using managed identity extension to retrieve access token" id="5222b31a-2198-4fb6-9c0b-90c9a42d5d14" I0227 18:41:34.179974 1 main.go:377] "checking credentials by getting authorizer" I0227 18:41:34.191144 1 plugins.go:43] Registered credential provider "akv2k8s" I0227 18:41:34.191176 1 auth.go:54] "auth service ca cert" file="/var/ca-cert/tls.crt" I0227 18:41:34.191184 1 auth.go:55] "auth service ca key" file="/var/ca-cert/tls.key" I0227 18:41:34.191282 1 main.go:393] "serving auth validation endpoint" path=":8080/auth/{namespace}/{pod}" I0227 18:41:34.191302 1 main.go:397] "serving health endpoint" path=":8080/healthz" I0227 18:41:34.191375 1 main.go:444] "serving encrypted auth endpoint" path=":9443/auth" 2023/02/27 18:41:34 [WARN] no tracer active I0227 18:41:34.191459 1 main.go:420] "serving encrypted webhook endpoint" path=":8443/pods" I0227 18:41:34.191479 1 main.go:423] "serving encrypted healthz endpoint" path=":8443/healthz" 2023/02/28 00:58:31 [DEBUG] reviewing request 84da1dd5-0965-46b4-b94d-b7f427fbe404, named: hgphs/ I0228 00:58:31.323023 1 main.go:142] "found pod to mutate" pod="hgphs/" I0228 00:58:31.323045 1 pod.go:285] "creating client certificate to use with auth service" hgphs/="(MISSING)" I0228 00:58:31.323052 1 clientCert.go:25] "creating x509 key pair for ca cert and key" I0228 00:58:31.323139 1 clientCert.go:32] "parse certificate" I0228 00:58:31.323163 1 clientCert.go:38] "generating client key" I0228 00:58:31.334819 1 clientCert.go:44] "generating serial number" I0228 00:58:31.334833 1 clientCert.go:66] "crating x509 certificate" I0228 00:58:31.336897 1 pod.go:292] "mutate init-containers" hgphs/="(MISSING)" I0228 00:58:31.336911 1 pod.go:298] "mutate containers" hgphs/="(MISSING)" I0228 00:58:31.336918 1 pod.go:116] "found container to mutate" container="hgphs/attphs-rasp-image" I0228 00:58:31.336925 1 pod.go:119] "checking for env vars to inject" container="hgphs/attphs-rasp-image" I0228 00:58:31.336936 1 pod.go:122] "found env var to inject" env="dr-staging-mysqlusername-secret-inject@azurekeyvault" container="hgphs/attphs-rasp-image" I0228 00:58:31.336948 1 pod.go:122] "found env var to inject" env="dr-staging-mysqlpasswd-secret-inject@azurekeyvault" container="hgphs/attphs-rasp-image" I0228 00:58:31.336958 1 pod.go:122] "found env var to inject" env="dr-staging-mysqlhost-secret-inject@azurekeyvault" container="hgphs/attphs-rasp-image" I0228 00:58:31.336973 1 pod.go:122] "found env var to inject" env="dr-staging-oidc-client-id-secret-inject@azurekeyvault" container="hgphs/attphs-rasp-image" I0228 00:58:31.336985 1 registry.go:30] "getting container command for container" container="hgphs/attphs-rasp-image" I0228 00:58:31.336999 1 registry.go:36] "no cmd override in kubernetes for container, checking docker image configuration for entrypoint and cmd" image="28363devopsacr.azurecr.io/attphs-rasp:4.3.4.557" container="hgphs/attphs-rasp-image" E0228 00:58:31.430423 1 main.go:165] "failed to mutate" err=< failed to get auto cmd, error: GET https://28363devopsacr.azurecr.io/oauth2/token?scope=repository%3Aattphs-rasp%3Apull&service=28363devopsacr.azurecr.io: UNAUTHORIZED: authentication required, visit https://aka.ms/acr/authorization for more information. cannot fetch image descriptor github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry.getImageConfig /go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry/registry.go:144 github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry.(Registry).GetImageConfig /go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry/registry.go:103 main.getContainerCmd /go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/registry.go:39 main.podWebHook.mutateContainers /go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/pod.go:143 main.podWebHook.mutatePodSpec /go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/pod.go:299 main.vaultSecretsMutator /go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/main.go:163 github.com/slok/kubewebhook/pkg/webhook/mutating.MutatorFunc.Mutate /go/pkg/mod/github.com/slok/kubewebhook@v0.11.0/pkg/webhook/mutating/mutator.go:25 github.com/slok/kubewebhook/pkg/webhook/mutating.mutationWebhook.mutatingAdmissionReview /go/pkg/mod/github.com/slok/kubewebhook@v0.11.0/pkg/webhook/mutating/webhook.go:128 github.com/slok/kubewebhook/pkg/webhook/mutating.mutationWebhook.Review /go/pkg/mod/github.com/slok/kubewebhook@v0.11.0/pkg/webhook/mutating/webhook.go:120 github.com/slok/kubewebhook/pkg/webhook/internal/instrumenting.(Webhook).Review /go/pkg/mod/github.com/slok/kubewebhook@v0.11.0/pkg/webhook/internal/instrumenting/instrumenting.go:42 github.com/slok/kubewebhook/pkg/http.HandlerFor.func1 /go/pkg/mod/github.com/slok/kubewebhook@v0.11.0/pkg/http/handler.go:64 net/http.HandlerFunc.ServeHTTP /usr/local/go/src/net/http/server.go:2109 github.com/gorilla/mux.(Router).ServeHTTP /go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210 net/http.serverHandler.ServeHTTP /usr/local/go/src/net/http/server.go:2947 net/http.(conn).serve /usr/local/go/src/net/http/server.go:1991 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1594
Additional context The configuration is largely default. so it's an out of the box install. There is no aad-pod-identity addon installed. It should be using the default AKS credentials. but again, unclear how to verify this. I've given Reader and acrPull to the aks cluster and as the output shows, i should be good???
$ az aks check-acr --name hgphs-westus2-dr-aks-01 --resource-group hgphs-westus2-dr-rg-01 --acr 28363devopsacr.azurecr.io Merged "hgphs-westus2-dr-aks-01" as current context in /tmp/tmpriic11sh WARNING: version difference between client (1.26) and server (1.24) exceeds the supported minor version skew of +/-1
[2023-02-28T01:34:47Z] Checking host name resolution (28363devopsacr.azurecr.io): SUCCEEDED [2023-02-28T01:34:47Z] Canonical name for ACR (28363devopsacr.azurecr.io): 28363devopsacr.privatelink.azurecr.io. [2023-02-28T01:34:47Z] Checking managed identity... [2023-02-28T01:34:47Z] Kubelet managed identity client ID: 5222b31a-2198-4fb6-9c0b-90c9a42d5d14 [2023-02-28T01:34:48Z] Validating managed identity existance: SUCCEEDED [2023-02-28T01:34:48Z] Validating image pull permission: SUCCEEDED [2023-02-28T01:34:48Z] Your cluster can pull images from 28363devopsacr.azurecr.io!