[Feature] Cache image signature and attestation

[honnix]

Problem Statement

It seems Kyverno does not cache image signature and attestation that are used to verify an image. For a particular image digest, under normal circumstances, the signature and attestation don't change once published (altering them afterwards does not see to be a reasonable and secure operation), therefore, having them cached would save the external calls made to registries.

There exists successful verification cache in Kyverno since 1.11, which makes the cache requested in this issue less useful; however for failed verification this cache would be very useful to avoid unnecessary network traffic that could substantially increase the mutation webhook latency, which in turn pulls down the whole k8s cluster performance.

Note that negative cache is a controversial feature request that is unlikely to be implemented, see #9503.

Solution Description

Signature and attestations associated with a specific image digest are cached in memory if enabled, otherwise keep existing behaviour for backward compatibility.

Alternatives

No response

Additional Context

No response

Slack discussion

No response

Research

• [X] I have read and followed the documentation AND the troubleshooting guide.
• [X] I have searched other issues in this repository and mine is not recorded.

[chipzoller]

I'm not clear how this issue is different from the behavior which is already in Kyverno 1.11 and what was raised in #9503.

[honnix]

Thank you for replying quickly!

Does Kyverno 1.11 cache signatures and attestations? If I read it correctly Kyverno offloads the signature and attestation fetching to cosign which does not do caching.

In a busy system with majority of images unsigned/unattested, Kyverno does not seem to scale well under our test and each mutation webhook call may take 1.5-2 seconds to finish. We believe the major issue is external calls made to registries. #9503 explored a negative cache to avoid verifying the same image over and over again. We understand that is controversial and may not be implemented, so this feature request takes a step back exploring the possibility avoiding making expensive network calls.

I hope this explains it more clearly.

[vishal-chdhry]

We do not cache signatures and cache image verification outcomes: https://kyverno.io/docs/writing-policies/verify-images/#cache

Also, one problem with storing signatures is that signatures can be deleted. In the case of Notary, signatures are just another artifact in the registry pointing at the image.

The reason why we do not cache failure is that a failure can happen because of several reasons that are out of the scope of image verification such as, bad auth credentials and connection issues and we don't want users to be stuck because of it.

[honnix]

@vishal-chdhry Yeah I understand there is positive cache and I also understand why you prefer not to cache signature/attestation by default. I think it might be OK to offer a switch so users can decide whether they would like to have a cache. I'm sure there are users having strict rules not to manipulate any existing signature/attestation, and for them the cache would be very useful to avoid having heavy traffic towards registries. Wdyt?

[chipzoller]

The scope here is substantially different. Caching "decisions about images" is different and easier than caching "signatures and attestations" of those images. The latter would probably require more thought into storage as some of these attestations can be quite large. So the implementation is unlikely to be a simple switch which, in turn, is unlikely to be a quick thing to produce. Doesn't invalidate the idea, but all of these revolve around a central theme of a more advanced image artifact cache which wasn't the goal of the current implementation.

[honnix]

The latter would probably require more thought into storage as some of these attestations can be quite large.

Sure. Using a cache is a typical storage vs. time choice to make. While users can play with number of cached items, overall cache size, and TTL to optimize memory footprint, it is nearly impossible for them to optimize registry reading network latency. So from this perspective, I still believe having such an option and mechanism is strictly better than not having.

[honnix]

https://github.com/sigstore/cosign/issues/1777#issuecomment-1282670046 seems to be related. I understand it is not straight forward to add the caching layer in Kyverno because the heavy duty is all done in:

cosign.VerifyImageSignatures(ctx, signedImgRef, co)
cosign.VerifyImageAttestations(ctx, signedImgRef, co)

where there is no way to inject a cache. Replicating the logic in those two functions in Kyverno also doesn't seem to be ideal because it will slowly diverge from cosign, plus some of the APIs in cosign are not opened.

[realshuting]

@vishal-chdhry - any updates?

[vishal-chdhry]

we cannot add caching to Cosign, but we can add it to notary and sigstore go