When enabling trust portieris errors "no valid ImagePullSecret defined for image"

mikednight commented 1 year ago

What commit ID of Portieris did you experience the problem with?

e4b12c28e5a2109c05dc91dc36c574b2752d8048

What went wrong?

When enabling trust on private repo and private notary portieris returns "no valid ImagePullSecret defined"

What should have happened differently?

Portieris should allow the image when trust, secret and repository are correctly configured

How can it be reproduced?

using jfrog for docker repo and private notary.

Any other relevant information

I am attempting to set up portieris with a private notary and jfrog. It works as expected, both allowing and denying images from the repository if trust is set. However, when i attempt to enforce trust, the logs complain about imagePullSecret not being found. The same imagePullSecret is defined in both the deployment that portieris is using and the deployment that portieris is blocking. I tried both types of docker secrets, auth and uname/password and could not work around the issue.

Setup:

2 separate namespaces, portieris and deployment
same secret in each namespace, mounted by portieris serviceaccount and directly in the deployment
clusterimagepolicy with trust set for private docker repo on jfrog and private notary
works as expected without setting trust

i've obfuscated our internal names for below clusterimage policy example:

  - name: docker-hub-internal/*
    policy:
      mutateImage: false
      trust:
        enabled: true
        trustServer: https://signing.internal

log outputs:

I0223 22:18:29.702292       1 controller.go:64] Processing admission request for CREATE on deployment
I0223 22:18:29.703909       1 controller.go:176] Getting policy for container image: docker-hub-internal/image:latest   namespace: provision
E0223 22:18:30.171868       1 controller.go:253] secret regcred not defined for registry: docker-hub-internal
I0223 22:18:30.177669       1 enforcer.go:95] policy.Trust {0xc00079369a [] https://signing.internal}
I0223 22:18:30.564048       1 responder.go:93] trust: policy denied the request: Deny "docker-hub-internal/image:latest", no valid ImagePullSecret defined for docker-hub-internal/image:latest
I0223 22:18:30.564073       1 controller.go:125] Deny for images:  [docker-hub-internal/image:latest]

molepigeon commented 1 year ago

Hi there. Thanks for your report. This line from your log is interesting:

secret regcred not defined for registry: docker-hub-internal

That is the only line that looks like that, which means it's the only secret that Portieris is seeing as in scope (ie either in the serviceaccount or directly mentioned in the resource spec).

Is regcred the secret that contains credentials for your registry? If so, I'm curious what the server part of the secret is. If you get the secret and decode the base64 content of the dockerconfigjson key, you should get a json object with a mapping of server names to user/passwords. Please share only the server name from that json object.

What's happening under the covers is that Portieris derives the server name from the image name (in this case docker-hub-internal) and is looking for a username/password for that specific server. If it doesn't find one it'll post the above log message. So for some reason it's not matching the server name in the secret. Does the server name in the secret have a DNS suffix that isn't in the image, for example?

mikednight commented 1 year ago

thanks for such quick feedback, i've double-checked everything here and it looks like it is correct. Obviously i have to remove some things for security reasons, but this is the config

k get secret regcred -o json | jq -r '.data.".dockerconfigjson"' | base64 -D | jq '.'
{
  "auths": {
    "https://docker-hub-internal": {
      "username": "username",
      "password": "password",
      "email": "test@email.com",
      "auth": "dXNlcm5hbWU6cGFzc3dvcmQK"
    }
  }
}

k get sa portieris -o yaml
apiVersion: v1
imagePullSecrets:
- name: regcred
kind: ServiceAccount
metadata:
  creationTimestamp: "2023-02-23T22:10:44Z"
  labels:
    app: portieris
    argocd.argoproj.io/instance: portieris
    chart: portieris-v0.13.2
    heritage: Helm
    release: portieris
  name: portieris
  namespace: security
  resourceVersion: "284384856"
  uid: 4811e8bc-ed77-4216-9062-48f8b3567595
secrets:
- name: portieris-token-r86fj

i've double checked that the secrets in both namespaces match, and likewise the deployment will rollout as expected if i don't have trust. I also tried removing "https" from the secret, but did not have an effect.

pre commented 7 months ago

Possibly related

https://github.com/IBM/portieris/issues/456

IBM / portieris