Deploying GitHub attested docker images to a private GKE cluster

[Navya-Teja-Otturu]

I have tried deploying a GitHub attested docker image stored in a Google Artifact registry in a GCP project to a private GKE cluster present in a different GCP project. The GKE cluster is configured with policy controller and trust policies helm charts by using the following commands.

helm upgrade policy-controller --install --atomic \
  --create-namespace --namespace artifact-attestations \
  oci://ghcr.io/github/artifact-attestations-helm-charts/policy-controller \
  --version v0.10.0-github5 \
  --set "webhook.serviceAccount.annotations.iam\.gke\.io/gcp-service-account=gh-policy-controller@gcpproject1.iam.gserviceaccount.com"

In the above command I have used a service account annotation so that policy controller can pull the manifests from the Google Artifact registry.

helm upgrade trust-policies --install --atomic --debug \
 --namespace artifact-attestations \
 oci://ghcr.io/github/artifact-attestations-helm-charts/trust-policies \
 --version v0.5.0 \
 --set policy.enabled=true \
 --set policy.organization=my-github-org \
 --set-json 'policy.exemptImages=["index.docker.io/library/busybox**"]' \
 --set-json 'policy.images=["europe-west1-docker.pkg.dev/**"]'

I have created a label on the target namespace in order to enforce the policies.

However, when I am trying to deploy to the target namespace using a workflow it fails with the following error.

Logs from GitHub workflow

Starting deploy...
 - configmap/test-gh-attestation-64tcfch2f9 unchanged
 - service/test-gh-attestation configured
 - Error from server (InternalError): error when applying patch:
 - {"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{},\"labels\":{\"provider\":\"waas\",\"repository.commit_sha\":\"xxxxxxxxx\",\"repository.name\":\"test-repo\",\"skaffold.dev/run-id\":\"da062413-17fb-4ba0-a882-4eb2d5a220d2\"},\"name\":\"test-gh-attestation\",\"namespace\":\"demo-pp\"},\"spec\":{\"replicas\":1,\"selector\":{\"matchLabels\":{\"service\":\"test-gh-attestation\"}},\"template\":{\"metadata\":{\"labels\":{\"provider\":\"waas\",\"repository.commit_sha\":\"xxxxxx\",\"repository.name\":\"test-repo\",\"service\":\"test-gh-attestation\",\"skaffold.dev/run-id\":\"da062413-17fb-4ba0-a882-4eb2d5a220d2\"}},\"spec\":{\"affinity\":{\"podAntiAffinity\":{\"preferredDuringSchedulingIgnoredDuringExecution\":[{\"podAffinityTerm\":{\"labelSelector\":{\"matchLabels\":{\"service\":\"test-gh-attestation\"}},\"topologyKey\":\"kubernetes.io/hostname\"},\"weight\":50}]}},\"containers\":[{\"envFrom\":[{\"configMapRef\":{\"name\":\"test-gh-attestation-64tcfch2f9\"}}],\"image\":\"europe-west1-docker.pkg.dev/registry/docker/test-gh-attestation@sha256:025abdf50674fa5d2ead4e5f1328f5e13a1e05e03582310a29377560c7c9db33\",\"imagePullPolicy\":\"IfNotPresent\",\"livenessProbe\":{\"failureThreshold\":3,\"httpGet\":{\"path\":\"/.well-known/live\",\"port\":8080,\"scheme\":\"HTTP\"},\"initialDelaySeconds\":60,\"periodSeconds\":10,\"successThreshold\":1,\"timeoutSeconds\":1},\"name\":\"test-gh-attestation\",\"ports\":[{\"containerPort\":8080,\"protocol\":\"TCP\"}],\"readinessProbe\":{\"failureThreshold\":3,\"httpGet\":{\"path\":\"/.well-known/ready\",\"port\":8080,\"scheme\":\"HTTP\"},\"initialDelaySeconds\":60,\"periodSeconds\":10,\"successThreshold\":1,\"timeoutSeconds\":1},\"resources\":{\"limits\":{\"cpu\":\"2000m\",\"memory\":\"1024Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"100Mi\"}}}]}}}}\n"},"labels":{"skaffold.dev/run-id":"da062413-17fb-4ba0-a882-4eb2d5a220d2"}},"spec":{"template":{"metadata":{"labels":{"skaffold.dev/run-id":"da062413-17fb-4ba0-a882-4eb2d5a220d2"}},"spec":{"$setElementOrder/containers":[{"name":"test-gh-attestation"}],"containers":[{"image":"europe-west1-docker.pkg.dev/registry/docker/test-gh-attestation@sha256:025abdf50674fa5d2ead4e5f1328f5e13a1e05e03582310a29377560c7c9db33","name":"test-gh-attestation","resources":{"limits":{"cpu":"2000m","memory":"1024Mi"}}}]}}}}
 - to:
 - Resource: "apps/v1, Resource=deployments", GroupVersionKind: "apps/v1, Kind=Deployment"
 - Name: "test-gh-attestation", Namespace: "demo-pp"
 - for: "STDIN": error when patching "STDIN": Internal error occurred: failed calling webhook "policy.sigstore.dev": failed to call webhook: Post "[https://webhook.artifact-attestations.svc:443/validations?timeout=30s](https://webhook.artifact-attestations.svc/validations?timeout=30s)": context deadline exceeded
kubectl apply: exit status 1

Logs from policy controller

{"level":"info","ts":"2024-08-19T10:03:30.413Z","logger":"policy-controller","caller":"webhook/admission.go:93","msg":"Webhook ServeHTTP request=&http.Request{Method:\"POST\", URL:(*url.URL)(0xc00143f8c0), Proto:\"HTTP/1.1\", ProtoMajor:1, ProtoMinor:1, Header:http.Header{\"Accept\":[]string{\"application/json, */*\"}, \"Accept-Encoding\":[]string{\"gzip\"}, \"Content-Length\":[]string{\"5001\"}, \"Content-Type\":[]string{\"application/json\"}, \"User-Agent\":[]string{\"kube-apiserver-admission\"}}, Body:(*http.body)(0xc0013cdbc0), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:5001, TransferEncoding:[]string(nil), Close:false, Host:\"webhook.artifact-attestations.svc:443\", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:\"10.64.0.10:35438\", RequestURI:\"/mutations?timeout=10s\", TLS:(*tls.ConnectionState)(0xc00158aa50), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), ctx:(*context.cancelCtx)(0xc001160dc0), pat:(*http.pattern)(0xc000b880c0), matches:[]string(nil), otherValues:map[string]string(nil)}","commit":"c294fd1"}
{"level":"info","ts":"2024-08-19T10:03:30.419Z","logger":"policy-controller","caller":"defaulting/defaulting.go:158","msg":"Kind: \"/v1, Kind=Pod\" PatchBytes: null","commit":"c294fd1","knative.dev/kind":"/v1, Kind=Pod","knative.dev/namespace":"demo-pp","knative.dev/name":"","knative.dev/operation":"CREATE","knative.dev/resource":"/v1, Resource=pods","knative.dev/subresource":"","knative.dev/userinfo":"system:serviceaccount:kube-system:replicaset-controller"}
{"level":"info","ts":"2024-08-19T10:03:30.420Z","logger":"policy-controller","caller":"webhook/admission.go:151","msg":"remote admission controller audit annotations=map[string]string(nil)","commit":"c294fd1","knative.dev/kind":"/v1, Kind=Pod","knative.dev/namespace":"demo-pp","knative.dev/name":"","knative.dev/operation":"CREATE","knative.dev/resource":"/v1, Resource=pods","knative.dev/subresource":"","knative.dev/userinfo":"system:serviceaccount:kube-system:replicaset-controller","admissionreview/uid":"83bc754f-e7d8-46e7-bab0-0fae4f4dc8dd","admissionreview/allowed":true,"admissionreview/result":"nil"}
{"level":"info","ts":"2024-08-19T10:03:30.427Z","logger":"policy-controller","caller":"webhook/admission.go:93","msg":"Webhook ServeHTTP request=&http.Request{Method:\"POST\", URL:(*url.URL)(0xc000710120), Proto:\"HTTP/1.1\", ProtoMajor:1, ProtoMinor:1, Header:http.Header{\"Accept\":[]string{\"application/json, */*\"}, \"Accept-Encoding\":[]string{\"gzip\"}, \"Content-Length\":[]string{\"5194\"}, \"Content-Type\":[]string{\"application/json\"}, \"User-Agent\":[]string{\"kube-apiserver-admission\"}}, Body:(*http.body)(0xc001516a80), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:5194, TransferEncoding:[]string(nil), Close:false, Host:\"webhook.artifact-attestations.svc:443\", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:\"10.64.0.10:35446\", RequestURI:\"/validations?timeout=30s\", TLS:(*tls.ConnectionState)(0xc00158abb0), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), ctx:(*context.cancelCtx)(0xc001161590), pat:(*http.pattern)(0xc000b88060), matches:[]string(nil), otherValues:map[string]string(nil)}","commit":"c294fd1"}
{"level":"warn","ts":"2024-08-19T10:05:29.404Z","logger":"policy-controller","caller":"webhook/validator.go:1264","msg":"Failed to validate at least one policy for europe-west1-docker.pkg.dev/registry/docker/test-gh-attestation@sha256:3f73a86814a8235f0244c64081dd12ba2eb3a740c248d8add95127a0b2809ca5 wanted 1 policies, only validated 0","commit":"c294fd1","knative.dev/kind":"/v1, Kind=Pod","knative.dev/namespace":"demo-pp","knative.dev/name":"test-gh-attestation-96778b754-v25cl","knative.dev/operation":"CREATE","knative.dev/resource":"/v1, Resource=pods","knative.dev/subresource":"","knative.dev/userinfo":"system:serviceaccount:kube-system:replicaset-controller"}
{"level":"error","ts":"2024-08-19T10:05:29.405Z","logger":"policy-controller","caller":"validation/validation_admit.go:183","msg":"Failed the resource specific validation","commit":"c294fd1","knative.dev/kind":"/v1, Kind=Pod","knative.dev/namespace":"demo-pp","knative.dev/name":"test-gh-attestation-96778b754-v25cl","knative.dev/operation":"CREATE","knative.dev/resource":"/v1, Resource=pods","knative.dev/subresource":"","knative.dev/userinfo":"system:serviceaccount:kube-system:replicaset-controller","stacktrace":"knative.dev/pkg/webhook/resourcesemantics/validation.validate\n\tknative.dev/pkg@v0.0.0-20230612155445-74c4be5e935e/webhook/resourcesemantics/validation/validation_admit.go:183\nknative.dev/pkg/webhook/resourcesemantics/validation.(*reconciler).Admit\n\tknative.dev/pkg@v0.0.0-20230612155445-74c4be5e935e/webhook/resourcesemantics/validation/validation_admit.go:79\nknative.dev/pkg/webhook.New.admissionHandler.func4\n\tknative.dev/pkg@v0.0.0-20230612155445-74c4be5e935e/webhook/admission.go:123\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2166\nnet/http.(*ServeMux).ServeHTTP\n\tnet/http/server.go:2683\nknative.dev/pkg/webhook.(*Webhook).ServeHTTP\n\tknative.dev/pkg@v0.0.0-20230612155445-74c4be5e935e/webhook/webhook.go:302\nknative.dev/pkg/network/handlers.(*Drainer).ServeHTTP\n\tknative.dev/pkg@v0.0.0-20230612155445-74c4be5e935e/network/handlers/drain.go:113\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:3137\nnet/http.(*conn).serve\n\tnet/http/server.go:2039"}
{"level":"info","ts":"2024-08-19T10:05:29.405Z","logger":"policy-controller","caller":"webhook/admission.go:151","msg":"remote admission controller audit annotations=map[string]string(nil)","commit":"c294fd1","knative.dev/kind":"/v1, Kind=Pod","knative.dev/namespace":"demo-pp","knative.dev/name":"test-gh-attestation-96778b754-v25cl","knative.dev/operation":"CREATE","knative.dev/resource":"/v1, Resource=pods","knative.dev/subresource":"","knative.dev/userinfo":"system:serviceaccount:kube-system:replicaset-controller","admissionreview/uid":"e8d88277-bdf3-4f12-8a63-7202fc31d232","admissionreview/allowed":false,"admissionreview/result":"&Status{ListMeta:ListMeta{SelfLink:,ResourceVersion:,Continue:,RemainingItemCount:nil,},Status:Failure,Message:validation failed: context was canceled before validation completed: ,Reason:BadRequest,Details:nil,Code:400,}"}

Could you please help me in resolving this? Please let me know if you need any additional information regarding the same.

Best Regards, Navya Teja Otturu

[sigurdfalk]

We are having the same issues in AKS

[trevrosen]

cc @codysoyland Apologies for the late response - we're investigating this.

[codysoyland]

@Navya-Teja-Otturu, sorry for the delay in responding to this issue. This should be fixed in the latest release, version v0.10.0-github8. I'm working on updating the docs to reference this version, but it is published now if you'd like to give it a try. Thank you for the detailed report!