[BUG] Injection fails in some cases when using multiple `imagePullSecrets` in a deployment manifest

[tspearconquest]

Note: Make sure to check out known issues (https://akv2k8s.io/troubleshooting/known-issues/) before submitting

Components and versions Select which component(s) the bug relates to with [X].

[ ] Controller, version: x.x.x (docker image tag) [x] Env-Injector (webhook), version: x.x.x (docker image tag) [ ] Other

Describe the bug When specifying multiple imagePullSecrets values, Kubernetes simply tries each secret in order until it gets a successful connection and is able to retrieve the image.

AKV2K8S appears to only try to use the first value listed. It should replicate what Kubernetes does.

To Reproduce Steps to reproduce the behavior:

• Setup akv2k8s namespace
• Create a docker registry secret in the akv2k8s namespace
• Deploy akv2k8s
• Setup a second namespace for a deployment which will use keyvault secrets
• Setup a docker registry secret in this new namespace
• Create a second docker registry secret with different credentials for the same registry to be used for pulling down a sidecar container from a different namespace in the same registry.
• Setup a deployment with an initContainer that uses keyvault secrets and pulls from one registry, and a normal container that pulls from a different registry
  • The deployment should have an initContainer and a normal container
  • Add 2 imagePullSecrets values, one for each docker registry secret in this namespace
  • IMPORTANT: To reproduce this, the first secret in the deployment manifest should be valid for the normal container only.

Actual behavior Pod is never created, replicaset events show the message in the logs section.

Expected behavior Pod should be created and image pulled. The credentials are correct, and I have devised a workaround for some situations but not all.

Logs If applicable, add logs to help explain your problem.

Warning FailedCreate 15s replicaset-controller Error creating: Internal error occurred: failed calling webhook "pods.env-injector.admission.spv.no": failed to call webhook: an error on the server ("{\"response\":{\"uid\":\"redacted\",\"allowed\":false,\"status\":{\"metadata\":{},\"status\":\"Failure\",\"message\":\"failed to get auto cmd, error: GET https://redacted/gitlab-agent/inject-secrets/manifests/v15.1.0-754feda5: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:redacted/gitlab-agent/inject-secrets Type:repository]]\\ncannot fetch image descriptor\\ngithub.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry.getImageConfig\\n\\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry/registry.go:144\\ngithub.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry.(*Registry).GetImageConfig\\n\\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry/registry.go:103\\nmain.getContainerCmd\\n\\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/registry.go:39\\nmain.podWebHook.mutateContainers\\n\\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/pod.go:143\\nmain.podWebHook.mutatePodSpec\\n\\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/pod.go:293\\nmain.vaultSecretsMutator\\n\\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/main.go:163\\ngithub.com/slok/kubewebhook/pkg/webhook/mutating.MutatorFunc.Mutate\\n\\t/go/pkg/mod/github.com/slok/kubewebhook@v0.11.0/pkg/webhook/mutating/mutator.go:25\\ngithub.com/slok/kubewebhook/pkg/webhook/mutating.mutationWebhook.mutatingAdmissionReview\\n\\t/go/pkg/mod/github.com/slok/kubewebhook@v0.11.0/pkg/webhook/mutating/webhook.go:128\\ngithub.com/slok/kubewebhook/pkg/webhook/mutating.mutationWebhook.Review\\n\\t/go/pkg/mod/github.com/slok/kubewebhook@v0.11.0/pkg/webhook/mutating/webhook.go:120\\ngithub.com/slok/kubewebhook/pkg/webhook/internal/instrumenting.(*Webhook).Review\\n\\t/go/pkg/mod/github.com/slok/kubewebhook@v0.11.0/pkg/") has prevented the request from succeeding

Additional context Add any other context about the problem here.

The workaround I devised is to switch the order of the secrets in the manifest for my deployment which is being injected.

I had this:

imagePullSecrets:
- name: gitlab-registry
- name: gitlab-registry-alt

And reversed it to this:

imagePullSecrets:
- name: gitlab-registry-alt
- name: gitlab-registry

The secrets in the akv2k8s namespace don't make a difference here, the secrets in the injected deployment's namespace are the ones being picked up by akv2k8s, but it seems to only be trying the first one and then giving up.

I haven't tested other scenarios as far as init containers vs normal container with/without secrets. The order of the values for imagePullSecrets in the deployment manifest should not matter here but currently it appears it does. Please let me know what other info I can provide.

[jeffwmiles]

I've encountered this issue today, and been trying to figure out what to do about it. Simply reversing the order of imagePullSecrets isn't valid for my use case.

One option we've got is to ensure the command is supplied within the container Spec in our K8s deployment. This is because the getContainerCmd function will skip the registry call since it doesn't need to find the Entrypoint anymore: https://github.com/SparebankenVest/azure-key-vault-to-kubernetes/blob/master/cmd/azure-keyvault-secrets-webhook/registry.go#L35

This does seem to be a problem in the underlying remote module, as opened here: https://github.com/google/go-containerregistry/issues/1431

A comment on that issue implies if the credentials are configured as specific as possible, rather than 2 secrets for the same destination, it should work. My use case is two different image Feeds within one artifact registry, so this might be possible.