search

nirmata / kyverno-notation-aws

Kyverno extension service for Notation and the AWS signer
Apache License 2.0
11 stars 11 forks source link

Error when using EKS Pod Identity as credentials provider #103

Open Bart-Leboeuf opened 10 months ago

Bart-Leboeuf commented 10 months ago

We've tried using EKS Pod Identity with kyervno-notation-aws (V1). It seems that the authentication endpoint is not taken into account by the application, so the role is not assumed. Using IRSA, it works correctly. In the same cluster, we have other containers that work perfectly with EKS Pod Identity.

Using IRSA :

2023-12-07T15:31:54.204Z    INFO    verifier/client.go:175  Token is authorized {TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name: GenerateName: Namespace: SelfLink: UID: ResourceVersion: Generation:0 CreationTimestamp:0001-01-01 00:00:00 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[] Finalizers:[] ManagedFields:[{Manager:kyverno-notation-aws Operation:Update APIVersion:authentication.k8s.io/v1 Time:2023-12-07 15:31:54 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:spec":{"f:token":{}}} Subresource:}]} Spec:{Token:eyJhbGciOiJSUzI1NiIsImtpZCI6IjQ5ODk4ZmUzYzQ1YmIyMWRhMjM5MzkyZGIxOTI0ZmUzYzVjNGQ2OWQifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjIl0sImV4cCI6MTczMzQ5NjQ2NSwiaWF0IjoxNzAxOTYwNDY1LCJpc3MiOiJodHRwczovL29pZGMuZWtzLmV1LXdlc3QtMy5hbWF6b25hd3MuY29tL2lkLzcwN0Q4NDc0OTREMUQ0ODA4QjI4RDYzQTJCRDBEQUY3Iiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJreXZlcm5vIiwicG9kIjp7Im5hbWUiOiJreXZlcm5vLXJlcG9ydHMtY29udHJvbGxlci1kOGI3ZDc0OTgtcXA4a24iLCJ1aWQiOiJhNzEyNmY3OC04ZWVkLTRkMGUtOWU2ZS05OTAyY2Q2NzI4ZTAifSwic2VydmljZWFjY291bnQiOnsibmFtZSI6Imt5dmVybm8tcmVwb3J0cy1jb250cm9sbGVyIiwidWlkIjoiNTg4ODY1OTItYzI5Ni00MmY4LWI0N2YtOGVjOWFlNWViZDZjIn0sIndhcm5hZnRlciI6MTcwMTk2NDA3Mn0sIm5iZiI6MTcwMTk2MDQ2NSwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt5dmVybm86a3l2ZXJuby1yZXBvcnRzLWNvbnRyb2xsZXIifQ.hFsAk9fKz60IOM7UxeCcIlUHEecbBZzg54LZW73wsUegutpynQujTUmS1wg3tcGKMIP6MMO_SLHJjVExaXXkKxhA-jIjGbX7bQz2mamuSaR171cmyWQOZ7XQTvu5D34Dw39836DGHOBv7BJqT_e6BTz31hnbk-N9ZJboj27MkBH9rRanbtdYqOLv40x6bIz41kUuRHH6OqfvAZ7_dk14bDbdM6X5srSwPm9P2oO2ojOl3hBKMtq7dXU_k5-WHYniPbXXix5wMJDgEyaxw1PCElD8AnG5ZwlksDkt-hKneGV5vIABIjF5Fqk96lrWcR8_2SUGR5g4h8y_DACEfSpctA Audiences:[]} Status:{Authenticated:true User:{Username:system:serviceaccount:kyverno:kyverno-reports-controller UID:58886592-c296-42f8-b47f-8ec9ae5ebd6c Groups:[system:serviceaccounts system:serviceaccounts:kyverno system:authenticated] Extra:map[authentication.kubernetes.io/pod-name:[kyverno-reports-controller-d8b7d7498-qp8kn] authentication.kubernetes.io/pod-uid:[a7126f78-8eed-4d0e-9e6e-9902cd6728e0]]} Audiences:[https://kubernetes.default.svc] Error:}}
2023-12-07T15:31:54.204Z    INFO    verifier/client.go:188  Request recieved with data={ImageReferences:[123456662101.dkr.ecr.eu-west-3.amazonaws.com*] Images:{InitContainers:map[] Containers:map[boweb-iam-frontend:{ImageInfo:{Registry:123456662101.dkr.ecr.eu-west-3.amazonaws.com Name:myaws1m-iam-frontend Path:myaws1m-iam-frontend Tag:0.1.6 Digest:} Pointer:/spec/template/spec/containers/0/image}] EphemeralContainers:map[]} TrustPolicy:mynet-trust-policy Attestations:[] Metadata:}
2023-12-07T15:31:54.204Z    INFO    notationfactory/client.go:91    Using trust policy provided in the request mynet-trust-policy
2023-12-07T15:31:54.204Z    INFO    notationfactory/client.go:102   Found notation verifer for trust policy mynet-trust-policy
2023-12-07T15:31:54.204Z    INFO    cache/client.go:120 Getting image from the cache: trustPolicy=mynet-trust-policy, imageRef=123456662101.dkr.ecr.eu-west-3.amazonaws.com/myaws1m-iam-frontend:0.1.6
2023-12-07T15:31:54.204Z    INFO    cache/client.go:139 Entry found in the cache mynet-trust-policy;123456662101.dkr.ecr.eu-west-3.amazonaws.com/myaws1m-iam-frontend:0.1.6 entry={{123456662101.dkr.ecr.eu-west-3.amazonaws.com myaws1m-iam-frontend myaws1m-iam-frontend 0.1.6 sha256:88d3093e15aa236813d14d346c8a0c0349459e35cc894dc86a2ad5a07cff3e32} /spec/containers/0/image}
2023-12-07T15:31:54.204Z    INFO    verifier/verify.go:359  Entry for the image found in cache, skipping image={{123456662101.dkr.ecr.eu-west-3.amazonaws.com myaws1m-iam-frontend myaws1m-iam-frontend 0.1.6 } /spec/template/spec/containers/0/image}; trustpolicy=mynet-trust-policy
2023-12-07T15:31:54.204Z    INFO    verifier/verify.go:125  verified map[%!d(string=myaws1m-iam-frontend):{{%!d(string=123456662101.dkr.ecr.eu-west-3.amazonaws.com) %!d(string=myaws1m-iam-frontend) %!d(string=myaws1m-iam-frontend) %!d(string=0.1.6) %!d(string=)} %!d(string=/spec/template/spec/containers/0/image)}] containers 
2023-12-07T15:31:54.204Z    INFO    verifier/verify.go:145  verified map[] initContainers
2023-12-07T15:31:54.204Z    INFO    verifier/verify.go:165  verified map[] ephemeralContainers
2023-12-07T15:31:54.204Z    INFO    verifier/response.go:109    building attestation set []
2023-12-07T15:31:54.204Z    INFO    verifier/verify.go:171  built attestation list%!(EXTRA map[string]types.AttestationList=map[123456662101.dkr.ecr.eu-west-3.amazonaws.com/myaws1m-iam-frontend:0.1.6:map[]])
2023-12-07T15:31:54.204Z    INFO    verifier/verify.go:181  verifying attestations map[123456662101.dkr.ecr.eu-west-3.amazonaws.com/myaws1m-iam-frontend:0.1.6:map[]]
2023-12-07T15:31:54.204Z    INFO    verifier/verify.go:191  verifying attestation, image=123456662101.dkr.ecr.eu-west-3.amazonaws.com/myaws1m-iam-frontend:0.1.6; attestations=map[]
2023-12-07T15:31:54.204Z    INFO    verifier/response.go:104    Sending response result=[{Operation:replace Path:/spec/containers/0/image Value:123456662101.dkr.ecr.eu-west-3.amazonaws.com/myaws1m-iam-frontend@sha256:88d3093e15aa236813d14d346c8a0c0349459e35cc894dc86a2ad5a07cff3e32}]
2023-12-07T15:31:54.204Z    INFO    verifier/client.go:226  Sending response {

Using Pod Identity association :

2023-12-07T15:49:43.093Z    INFO    verifier/client.go:175  Token is authorized {TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name: GenerateName: Namespace: SelfLink: UID: ResourceVersion: Generation:0 CreationTimestamp:0001-01-01 00:00:00 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[] Finalizers:[] ManagedFields:[{Manager:kyverno-notation-aws Operation:Update APIVersion:authentication.k8s.io/v1 Time:2023-12-07 15:49:43 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:spec":{"f:token":{}}} Subresource:}]} Spec:{Token:eyJhbGciOiJSUzI1NiIsImtpZCI6IjQ5ODk4ZmUzYzQ1YmIyMWRhMjM5MzkyZGIxOTI0ZmUzYzVjNGQ2OWQifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjIl0sImV4cCI6MTczMzQ5OTQzMywiaWF0IjoxNzAxOTYzNDMzLCJpc3MiOiJodHRwczovL29pZGMuZWtzLmV1LXdlc3QtMy5hbWF6b25hd3MuY29tL2lkLzcwN0Q4NDc0OTREMUQ0ODA4QjI4RDYzQTJCRDBEQUY3Iiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJreXZlcm5vIiwicG9kIjp7Im5hbWUiOiJreXZlcm5vLXJlcG9ydHMtY29udHJvbGxlci1kOGI3ZDc0OTgtcXA4a24iLCJ1aWQiOiJhNzEyNmY3OC04ZWVkLTRkMGUtOWU2ZS05OTAyY2Q2NzI4ZTAifSwic2VydmljZWFjY291bnQiOnsibmFtZSI6Imt5dmVybm8tcmVwb3J0cy1jb250cm9sbGVyIiwidWlkIjoiNTg4ODY1OTItYzI5Ni00MmY4LWI0N2YtOGVjOWFlNWViZDZjIn0sIndhcm5hZnRlciI6MTcwMTk2NzA0MH0sIm5iZiI6MTcwMTk2MzQzMywic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt5dmVybm86a3l2ZXJuby1yZXBvcnRzLWNvbnRyb2xsZXIifQ.KX6KzUAfW4hD6mQRKqlCg6yU16hvtW71w-C8TMqZRh7ti--Gn5EDyI6O3Za-4dqANS8t6ezB1BHd7TwIXEcBsxibFWoBc72eb5cjAOK5RcC2sy-7jCi0jpxXQ3D8dJD5dibVCiwaeCUs017SVnd9rIcYrfng6ClBx8Jp02s2cOdx619MGvQM-taoF7Xtp3e-wG7hzPly8kqWXY4UZWqRBvRUZCvNR4ISjGm9XlWDqeYPkMLzSCjr-LjOQN3Ou5ZnrZeEcmKMDYbgVAz3Y3v4RsZKGj2CAV2lGyzr3-SirHP6wqNjSbg Audiences:[]} Status:{Authenticated:true User:{Username:system:serviceaccount:kyverno:kyverno-reports-controller UID:58886592-c296-42f8-b47f-8ec9ae5ebd6c Groups:[system:serviceaccounts system:serviceaccounts:kyverno system:authenticated] Extra:map[authentication.kubernetes.io/pod-name:[kyverno-reports-controller-d8b7d7498-qp8kn] authentication.kubernetes.io/pod-uid:[a7126f78-8eed-4d0e-9e6e-9902cd6728e0]]} Audiences:[https://kubernetes.default.svc] Error:}}
2023-12-07T15:49:43.093Z    INFO    verifier/client.go:188  Request recieved with data={ImageReferences:[123456662101.dkr.ecr.eu-west-3.amazonaws.com*] Images:{InitContainers:map[] Containers:map[boweb-iam-frontend:{ImageInfo:{Registry:123456662101.dkr.ecr.eu-west-3.amazonaws.com Name:myaws1m-iam-frontend Path:myaws1m-iam-frontend Tag:0.1.6 Digest:} Pointer:/spec/template/spec/containers/0/image}] EphemeralContainers:map[]} TrustPolicy:mynet-trust-policy Attestations:[] Metadata:}
2023-12-07T15:49:43.093Z    INFO    notationfactory/client.go:91    Using trust policy provided in the request mynet-trust-policy
2023-12-07T15:49:43.093Z    INFO    notationfactory/client.go:102   Found notation verifer for trust policy mynet-trust-policy
2023-12-07T15:49:43.093Z    INFO    cache/client.go:120 Getting image from the cache: trustPolicy=mynet-trust-policy, imageRef=123456662101.dkr.ecr.eu-west-3.amazonaws.com/myaws1m-iam-frontend:0.1.6
2023-12-07T15:49:43.093Z    ERROR   cache/client.go:130 Entry not found key=mynet-trust-policy;123456662101.dkr.ecr.eu-west-3.amazonaws.com/myaws1m-iam-frontend:0.1.6
2023-12-07T15:49:43.093Z    INFO    verifier/verify.go:362  Entry not found in the cache verifying image=123456662101.dkr.ecr.eu-west-3.amazonaws.com/myaws1m-iam-frontend:0.1.6
2023-12-07T15:49:43.093Z    INFO    verifier/verify.go:364  verifying image infos {ImageInfo:{Registry:123456662101.dkr.ecr.eu-west-3.amazonaws.com Name:myaws1m-iam-frontend Path:myaws1m-iam-frontend Tag:0.1.6 Digest:} Pointer:/spec/template/spec/containers/0/image}
2023-12-07T15:49:43.093Z    INFO    verifier/verify.go:382  verifying image 123456662101.dkr.ecr.eu-west-3.amazonaws.com/myaws1m-iam-frontend:0.1.6
2023-12-07T15:49:43.093Z    ERROR   verifier/verify.go:367  verification failed for image {{123456662101.dkr.ecr.eu-west-3.amazonaws.com myaws1m-iam-frontend myaws1m-iam-frontend 0.1.6 } /spec/template/spec/containers/0/image}: failed to resolve digest: failed to retrieve credentials: failed to load default configuration: invalid endpoint host, "169.254.170.23", only loopback hosts are allowed
2023-12-07T15:49:43.093Z    ERROR   verifier/verify.go:116  failed to verify container myaws1m-iam-frontend: failed to verify image {{123456662101.dkr.ecr.eu-west-3.amazonaws.com myaws1m-iam-frontend myaws1m-iam-frontend 0.1.6 } /spec/template/spec/containers/0/image}: failed to resolve digest: failed to retrieve credentials: failed to load default configuration: invalid endpoint host, "169.254.170.23", only loopback hosts are allowed
2023-12-07T15:49:43.093Z    ERROR   verifier/response.go:79 Verification failed with error failed to verify container myaws1m-iam-frontend: failed to verify image {{123456662101.dkr.ecr.eu-west-3.amazonaws.com myaws1m-iam-frontend myaws1m-iam-frontend 0.1.6 } /spec/template/spec/containers/0/image}: failed to resolve digest: failed to retrieve credentials: failed to load default configuration: invalid endpoint host, "169.254.170.23", only loopback hosts are allowed

I can see the credentials varaibles set on the Pod : 

apiVersion: v1
kind: Pod
metadata:
  name: kyverno-notation-aws-78d96bcc75-6qhxk
  generateName: kyverno-notation-aws-78d96bcc75-
  namespace: kyverno-notation-aws
  uid: 37cbb3ce-ed4d-4c0d-992e-6ffe1fda7502
  resourceVersion: '66079498'
  creationTimestamp: '2023-12-07T15:47:58Z'
  labels:
    app: kyverno-notation-aws
    pod-template-hash: 78d96bcc75
  selfLink: >-
    /api/v1/namespaces/kyverno-notation-aws/pods/kyverno-notation-aws-78d96bcc75-6qhxk
spec:
  volumes:
    - name: aws-iam-token
      projected:
        sources:
          - serviceAccountToken:
              audience: pods.eks.amazonaws.com
              expirationSeconds: 86400
              path: token
        defaultMode: 420
    - name: notation
      emptyDir: {}
    - name: kube-api-access-dxfz2
      projected:
        sources:
          - serviceAccountToken:
              expirationSeconds: 3607
              path: token
          - configMap:
              name: kube-root-ca.crt
              items:
                - key: ca.crt
                  path: ca.crt
          - downwardAPI:
              items:
                - path: namespace
                  fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
        defaultMode: 420
  containers:
    - name: kyverno-notation-aws
      image: >-
        ghcr.io/nirmata/kyverno-notation-aws:latest
      args:
        - '--debug'
        - '--cacheEnabled'
        - '--cacheMaxSize=2000'
        - '--cacheTTLDurationSeconds=7200'
      env:
        - name: NOTATION_DIR
          value: /notation
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: SERVICE_NAME
          value: svc
        - name: DEPLOYMENT_NAME
          value: kyverno-notation-aws
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: AWS_REGION
          value: eu-west-3
        - name: DEFAULT_TRUST_POLICY
          value: aws-signer-trust-policy
        - name: AWS_STS_REGIONAL_ENDPOINTS
          value: regional
        - name: AWS_CONTAINER_CREDENTIALS_FULL_URI
          value: http://169.254.170.23/v1/credentials
        - name: AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE
          value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
      resources:
        limits:
          memory: 512Mi
        requests:
          cpu: 100m
          memory: 32Mi
      volumeMounts:
        - name: notation
          mountPath: /notation
        - name: kube-api-access-dxfz2
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        - name: aws-iam-token
          readOnly: true
          mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      imagePullPolicy: Always
      securityContext:
        capabilities:
          drop:
            - ALL
        runAsUser: 2000
        runAsGroup: 3000
        allowPrivilegeEscalation: false
        seccompProfile:
          type: RuntimeDefault
  restartPolicy: Always
  terminationGracePeriodSeconds: 5
  dnsPolicy: ClusterFirst
  serviceAccountName: kyverno-notation-aws
  serviceAccount: kyverno-notation-aws
  securityContext:
    runAsNonRoot: true
  priority: 0
  enableServiceLinks: true
  preemptionPolicy: PreemptLowerPriority
calvinbui commented 3 days ago

it should work now that aws-sdk v2 has been upgraded.

@vishal-chdhry is the latest untagged tag safe to use? we are still using the commit that corresponds to the v1 tag.

vishal-chdhry commented 3 days ago

@calvinbui yes you can use latest, it is safe to use

calvinbui commented 2 days ago

thanks @vishal-chdhry the latest tag latest@sha256:c8ee5afd88cb1d6c4f0d27c9fb5581982841ca1ad9be742a1095cdcb89de60cc works - but eks pod identity is still not working. same error as OP listed.

failed to execute the verify-signature command for plugin com.amazonaws.signer.notation.plugin: ERROR: invalid endpoint host, "169.254.170.23", only loopback hosts are allowed

i believe the problem is the aws-signer plugin's dependency is out of date based on the error message. the version of their website is 1.0.298, GitHub is 1.0.350. i'll test this out.

vishal-chdhry commented 2 days ago

@calvinbui I think you are right The minimum SDK version for golang is a release from November 2023: https://docs.aws.amazon.com/eks/latest/userguide/pod-id-minimum-sdk.html

The signer binary was last updated in June 2023: Bin Download Page Changelog: https://d2hvyiie56hcat.cloudfront.net/CHANGELOG