[vault-secrets-webhook] pod mutation with multiple imagePullSecrets if first secret not found

[alexey-ban]

Describe the bug: K8s is able to pull images using the provided pull secrets, even if one of them doesn't exist

vault-secrets-webhook picks the first listed imagePullSecret and broke if it not found

Expected behaviour: vault-secrets-webhook should attempt all listed imagePullSecrets to find the one that works.

Steps to reproduce the bug:

1. Create deployment with 2 imagePullSecrets:

   imagePullSecrets:
   - name: registry1
   - name: registry2

2. Create only second (registry2) with valid credentials

Additional context: Add any other context about the problem here.

Environment details:

• Kubernetes version (e.g. v1.10.2): v1.18.9-eks-d1db3c
• Cloud-provider/provisioner (e.g. AKS, GKE, EKS, PKE etc): EKS
• bank-vaults version (e.g. 0.4.17): 1.11.3
• Install method (e.g. helm or static manifests): helm
• Logs from the misbehaving component (and any other relevant logs):

  time="2021-02-22T21:21:03Z" level=warning msg="no tracer active" app=vault-secrets-webhook
  time="2021-02-22T21:21:03Z" level=warning msg="no tracer active" app=vault-secrets-webhook
  time="2021-02-22T21:21:03Z" level=info msg="Listening on https://:8443" app=vault-secrets-webhook
  time="2021-02-22T21:21:40Z" level=error msg="admission webhook error: secrets \"registry1\" not found" app=vault-secrets-webhook

• Resource definition (possibly in YAML format) that caused the issue, without sensitive data:

/kind bug

[bonifaido]

Hi @alexey-ban, which version of the webhook do you use?

[alexey-ban]

Hi @bonifaido helm chart: 1.11.3, image: ghcr.io/banzaicloud/vault-secrets-webhook:1.11.2

[bonifaido]

Hi, I just found that we have an upstream issue here in the go-containerregistry library that checks the container registries: https://github.com/google/go-containerregistry/issues/723

[dkulchinsky]

Hey folks, We're struggling with a similar issue that was originally reported in https://github.com/banzaicloud/bank-vaults/issues/1232

In our case, we have multiple imagePullSecrets defined that indeed exist and are correct (the reason is because same Pod has containers coming from different internal repos that have different secrets).

How in this scenario, VSW is only able to pull images for the registry whose secret is the first in the imagePullSecrets list, if the image is in the registry whose secret is 2nd - it fails to pull the image and mutation is not performed.

[dkulchinsky]

Hey @bonifaido 👋🏼

I realize this is an upstream issue and looks like not something that can be easily fixed (although the PR that introduced the "bug" is really strange).

We are impacted by this due to our internal container registry having different secrets for different projects (namespace) in the registry.

Do you think there's anything that can be done in VSW to address this?

[csatib02]

Hey,

Can someone verify that this issue still present as of today? It seems like there has been some fixes around this one.