Open kartheek-cervello opened 1 year ago
I have tested the env_injector with the workload identity support and I do not have the problem.
UNAUTHORIZED: authentication required
do you have access to the image? Maybe the imagePullSecret or permission is missing?
@kartheek-cervello on azure you can use az aks check-acr
to check the access.
My helm values:
controller:
image:
tag: 1.5.0-beta.1
logLevel: info
podLabels:
azure.workload.identity/use: "true"
serviceAccount:
annotations:
azure.workload.identity/client-id: <myid>
create: true
labels:
azure.workload.identity/use: "true"
env_injector:
enabled: true
envImage:
tag: 1.5.0-beta.1
image:
tag: 1.5.0-beta.1
podLabels:
azure.workload.identity/use: "true"
serviceAccount:
annotations:
azure.workload.identity/client-id: <myid>
create: true
labels:
azure.workload.identity/use: "true"
name: akv2k8s-envinjector
global:
keyVaultAuth: environment-azidentity
My manifests:
apiVersion: v1
kind: Pod
metadata:
name: test
spec:
containers:
- image: mytest.azurecr.io/alpine:3.17
imagePullPolicy: Always
command:
- env
name: test
env:
- name: SECRET_NAME
value: "test-username-inject@azurekeyvault"
---
apiVersion: spv.no/v1
kind: AzureKeyVaultSecret
metadata:
name: test-username-inject
namespace: test
spec:
vault:
name: mykeyvault
object:
name: test-username
type: secret
Pod output:
test I0202 12:08:05.597938 1 version.go:31] "version info" version="" commit="79d3e52" buildDate="2022-12-28T20:31:03Z" component="vaultenv"
test I0202 12:08:05.598029 1 main.go:185] "azure key vault env injector initializing"
test I0202 12:08:05.598209 1 main.go:254] "found original container command" cmd="/usr/bin/env" args=[env]
test I0202 12:08:05.598262 1 authentication.go:110] "checking if current auth service credentials are stale" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/test/test?secret=akv2k8s-test"
test I0202 12:08:05.680814 1 authentication.go:123] "auth service credentials ok" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/test/test?secret=akv2k8s-test"
test I0202 12:08:05.681355 1 authentication.go:159] "requesting azure key vault oauth token" url="https://akv2k8s-envinjector.akv2k8s.svc:9443/auth/test/test"
test I0202 12:08:05.707612 1 authentication.go:179] "successfully received oauth token"
test I0202 12:08:05.846204 1 main.go:353] "secret injected into env var" azurekeyvaultsecret="test/test-username-inject" env="SECRET_NAME"
test I0202 12:08:05.846280 1 main.go:359] "starting process with secrets in env vars" cmd="/usr/bin/env" args=[env]
...
test SECRET_NAME=myusername
...
Stream closed EOF for test/test (test)
Stream closed EOF for test/test (copy-azurekeyvault-env)
@kartheek-cervello @cgroschupp We have a bug indeed... your example does not trigger this scenario. It only triggers when the container command is not set, when we need to query the container registry to get the ENTRYPOINT or CMD from the manifest https://github.com/SparebankenVest/azure-key-vault-to-kubernetes/blob/master/cmd/azure-keyvault-secrets-webhook/registry.go#L35
Can confirm I am experiencing the same issue using chart 2.3.1 on a clean install. Downgrading to 2.2.2 and it works fine.
Also confirmed resolution with version 2.2.2 of the chart.
@cgroschupp reference your comment you are using a beta release version of the image.
Are users/environments expected to use a pre-release version in production environments to make this work again?
@wsmelton The Workload Identity feature is currently not released, if you want to test it you will need to use the pre-release.
@kartheek-cervello Can you try what I mentioned here?
@cgroschupp and @181192 now that Workload Identity is GA Is there an ETA for a release?
Note: Make sure to check out known issues (https://github.com/sparebankenvest/azure-key-vault-to-kubernetes#known-issues) before submitting
Your question Unable to pull images from Azure container registry when using workload identity and akv2k8s together. We have updated the chart version to 2.3.1 and made necessary changes to CRDs as per #442
To Reproduce helm chart configuration for akv2k8s
controller.image: tag: 1.5.0-beta.1 controller.podLabels: | "azure.workload.identity/use": "true" controller.serviceAccount.annotations: | "azure.workload.identity/client-id": "xxxxxxxxx" controller.serviceAccount.labels: | "azure.workload.identity/use": "true" env_injector.envImage.tag: 1.5.0-beta.1 env_injector.image.tag: 1.5.0-beta.1 env_injector.podLabels: | "azure.workload.identity/use": "true" env_injector.serviceAccount.annotations: | "azure.workload.identity/client-id": "xxxxxxx" env_injector.serviceAccount.labels: | "azure.workload.identity/use": "true" global.keyVaultAuth: environment-azidentity
manifest file of the POD
deploy a pod that references the service account
apiVersion: v1 kind: Pod metadata: name: quick-start namespace: sndbx-2-eus2 spec: serviceAccountName: ksf-sndbx-2-eus2-agentpool containers:
kvmap manifest file apiVersion: spv.no/v2beta1 kind: AzureKeyVaultSecret metadata: name: secret-name namespace: sndbx-2-eus2 spec: vault: name: xxxxxxxxx #Key Vault Name object: name: SECRET-NAME # Name of Secret type: secret # Type of object: key, secret, certificate
Logs Error from server (InternalError): error when creating "demo.yaml": Internal error occurred: failed calling webhook "pods.env-injector.admission.spv.no": failed to call webhook: an error on the server ("{\"response\":{\"uid\":\"80361cd7-cf79-41f8-acaa-b20405c3890b\",\"allowed\":false,\"status\":{\"metadata\":{},\"status\":\"Failure\",\"message\":\"failed to get auto cmd, error: GET https://xxx.azurecr.io/oauth2/token?scope=repository%3Apact%2Fdev%2Fapp%3Apull\\u0026service=xxx.azurecr.io: UNAUTHORIZED: authentication required, visit https://aka.ms/acr/authorization for more information.\ncannot fetch image descriptor\ngithub.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry.getImageConfig\n\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry/registry.go:144\ngithub.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry.(Registry).GetImageConfig\n\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry/registry.go:103\nmain.getContainerCmd\n\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/registry.go:39\nmain.podWebHook.mutateContainers\n\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/pod.go:143\nmain.podWebHook.mutatePodSpec\n\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/pod.go:299\nmain.vaultSecretsMutator\n\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/main.go:163\ngithub.com/slok/kubewebhook/pkg/webhook/mutating.MutatorFunc.Mutate\n\t/go/pkg/mod/github.com/slok/kubewebhook@v0.11.0/pkg/webhook/mutating/mutator.go:25\ngithub.com/slok/kubewebhook/pkg/webhook/mutating.mutationWebhook.mutatingAdmissionReview\n\t/go/pkg/mod/github.com/slok/kubewebhook@v0.11.0/pkg/webhook/mutating/webhook.go:128\ngithub.com/slok/kubewebhook/pkg/webhook/mutating.mutationWebhook.Review\n\t/go/pkg/mod/github.com/slok/kubewebhook@v0.11.0/pkg/webhook/mutating/webhook.go:120\ngithub.com/slok/kubewebhook/pkg/webhook/internal/instrumenting.(Webhook).Review\n\t/go/pkg/mod/github.com/slok/kubewebhook@v0.11.0/pkg/webhook/internal/inst") has prevented the request from succeeding