Not able to verify container image in my k8s cluster

[mandeepgoyat]

Hi Team

I would like to verify my container images before deploying to pod in my k8s cluster. I am trying to achieve this with cosign-gatekeeper-provider i.e https://github.com/sigstore/cosign-gatekeeper-provider I have signed the image using cosign but I am not able to verify images when I try to deploy pod using manifest in my cluster.

I am getting following error:- error when creating “policy/examples/valid.yaml”: admission webhook “validation.gatekeeper.sh” denied the request: [cosign-gatekeeper-provider] invalid response: {“errors”: [], “responses”: [], “status_code”: 200, “system_error”: “VerifyImageSignatures: one of verifier or root certs is required”}

I am able to verify container images using commands on my local command line i.e cosign verify .(But not inside k8s cluster). Looks like keys/cert information are not propagated to cluster

Anything else you would like to add: I am not sure whether I should put this issue here or on cosign.

Environment:

Gatekeeper version:3.8.1

[ritazh]

Can you make sure the root cert exists?

if co.RootCerts == nil && co.SigVerifier == nil {
    return nil, false, errors.New("one of verifier or root certs is required")
}

https://github.com/sigstore/cosign/blob/0605cedf7dc2db66b9dfc3d4329c7169131908bc/pkg/cosign/verify.go#L367-L371

[mandeepgoyat]

Can you make sure the root cert exists?

if co.RootCerts == nil && co.SigVerifier == nil {
  return nil, false, errors.New("one of verifier or root certs is required")
}

https://github.com/sigstore/cosign/blob/0605cedf7dc2db66b9dfc3d4329c7169131908bc/pkg/cosign/verify.go#L367-L371

@ritazh Thanks for reply. You are right, key/cert information seems missing.. Thats what i mentioned in my query. If I perform cosign verify on my command prompt , it works but when i deploy the image via pod definition file inside cluster, it fail with above mentioned error. Any idea how to propagate key information to cluster

[rohrit]

I am getting the same error when I was trying this out. I checked the logs for the provider and I see errors related to fetching TUF repo data from the first attempt to install the policy/example/valid.yaml manifest. Subsequent attempts give the error related to verifier certs shown above in the original comment.

kubectl logs -f -l run=cosign-gatekeeper-provider -n cosign-gatekeeper-provider
starting server...
verify signature for: devopps/alpine:signed
2022/06/13 05:54:56 http: panic serving 172.17.0.1:21118: creating root cert pool: retrieving trusted root; local cache may be corrupt: updating from remote TUF repository: updating tuf metadata: tuf: unexpected file size: root.json (expected 5297 bytes, got 4810 bytes)
goroutine 34 [running]:
net/http.(*conn).serve.func1()
    /usr/local/go/src/net/http/server.go:1801 +0xb9
panic({0x1b9fa40, 0xc000955d58})
    /usr/local/go/src/runtime/panic.go:1047 +0x266
github.com/sigstore/cosign/cmd/cosign/cli/fulcio/fulcioroots.initRoots()
    /go/pkg/mod/github.com/sigstore/cosign@v1.3.1/cmd/cosign/cli/fulcio/fulcioroots/fulcioroots.go:67 +0x235
github.com/sigstore/cosign/cmd/cosign/cli/fulcio/fulcioroots.Get.func1()
    /go/pkg/mod/github.com/sigstore/cosign@v1.3.1/cmd/cosign/cli/fulcio/fulcioroots/fulcioroots.go:45 +0x17
sync.(*Once).doSlow(0xc0003de438, 0x15)
    /usr/local/go/src/sync/once.go:68 +0xd2
sync.(*Once).Do(...)
    /usr/local/go/src/sync/once.go:59
github.com/sigstore/cosign/cmd/cosign/cli/fulcio/fulcioroots.Get()
    /go/pkg/mod/github.com/sigstore/cosign@v1.3.1/cmd/cosign/cli/fulcio/fulcioroots/fulcioroots.go:44 +0x31
github.com/sigstore/cosign/cmd/cosign/cli/fulcio.GetRoots(...)
    /go/pkg/mod/github.com/sigstore/cosign@v1.3.1/cmd/cosign/cli/fulcio/fulcio.go:197
main.validate({0x218ee50, 0xc0008b8000}, 0xc0000e4600)
    /go/src/github.com/developer-guy/cosign-gatekeeper-provider/provider.go:72 +0x408
net/http.HandlerFunc.ServeHTTP(0x7f7c2bd3f2f8, {0x218ee50, 0xc0008b8000}, 0xc0008b8000)
    /usr/local/go/src/net/http/server.go:2046 +0x2f
net/http.(*ServeMux).ServeHTTP(0x0, {0x218ee50, 0xc0008b8000}, 0xc0000e4600)
    /usr/local/go/src/net/http/server.go:2424 +0x149
net/http.serverHandler.ServeHTTP({0x2182d08}, {0x218ee50, 0xc0008b8000}, 0xc0000e4600)
    /usr/local/go/src/net/http/server.go:2878 +0x43b
net/http.(*conn).serve(0xc000140320, {0x2199b40, 0xc00084be00})
    /usr/local/go/src/net/http/server.go:1929 +0xb08
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:3033 +0x4e8
verify signature for: devopps/alpine:signed
one of verifier or root certs is required

[Anna-Katona]

Any plans to add an opportunity to pass they key to the deployment?

[mathieu-benoit]

Just heads up that the errors creating root cert pool and one of verifier or root certs is required are now fixed in main branch (#26). You need to build your own container image with that.

With that said, I'm still Not able to verify container image in my k8s cluster too, even with that fix.

We are making progress let's say, but it's not yet solving the issue.

Don't know if someone can help in clarifying which scenario is covered/supported with this repo.

What's explained in there https://github.com/sigstore/cosign-gatekeeper-provider#verification was (and is still) not working.