Changing base images to cgr.dev/chainguard/static

[imjasonh]

Knative Serving and Eventing's container images are currently based on gcr.io/distroless/static:nonroot (serving, eventing).

This is a good thing. The static:nonroot image contains almost nothing, it runs as a nonroot user, and it's served Knative well since the early days. However, I'd like to discuss updating this default base image to cgr.dev/chainguard/static.

This new image is built using apko instead of GCP-distroless's Bazel build system, which means it's quite a bit easier to rebuild the image yourself, and easier to modify the image if there's ever such a need. This simpler development story also makes it easier to support, which means you'll get better support should anything come up.

This new image also runs as a nonroot user by default. This image is keylessly signed (as GCP-distroless is), and comes with an SBOM generated at build-time -- but since there's almost nothing in the image, there's almost nothing in the SBOM.

This new image supports all the same architectures as the current base, as well as linux/arm/v7, linux/riscv64 and linux/386.

It's also just a bit smaller -- from 783KB down to 479KB -- but that's probably not a big concern (pun intended!).

Tekton changed to this image a couple releases ago in https://github.com/tektoncd/pipeline/pull/5009, with no technical issues -- they also use an image that provides git, and another that provides busybox, both also switched to be built using apko with no issues.

ko switched to use this base image by default in v0.12, again with no technical issues that I'm aware of. Knative didn't pick up the ko change because you set your own default to gcr.io/distroless/static:nonroot in the links above.

If the community decides not to make this change, that's okay -- the current image is also a great choice. I just wanted to bring it up in case there was an appetite.

If there's interest, I can send PRs to Serving and Eventing, and any other relevant projects that are interested.

cc @mattmoor @vaikas for visibility

[mattmoor]

FWIW we've been using these in our hakn repackaging of knative serving without any issues for some time now.

[psschwei]

Other than having to remember to pass --sbom=none (I use Quay as my ko repo), I didn't run into any issues using it as part of my local dev workflow for Serving. Assuming no one has strong objections, I'd be ok with making this change.

[imjasonh]

cc @knative/technical-oversight-committee for visibility

[dprotaso]

My only concern is the use of the cgr.dev domain. Do you have mirrors of these images on public registries?

[imjasonh]

My only concern is the use of the cgr.dev domain. Do you have mirrors of these images on public registries?

That's understandable.

We don't currently have mirrors onto other registries, but might consider it.

The specific concern would be that if cgr.dev is down, any releases or e2e tests of Knative would be broken until it becomes available again, correct? If so, some of that could be alleviated by having a separate process mirror cgr.dev/chainguard/static into gcr.io/knative-releases/... (and its SBOMs, signatures, etc.), and then basing Knative's images on that copy. That at least gives you a fallback in case cgr.dev is unavailable, at the cost of a bit more automation to handle that mirroring. If that's something you'd like to explore I can help with that too.

In any case, once Knative builds its images the reachability of cgr.dev is moot, since Knative serves all its image contents from gcr.io.

[dprotaso]

The specific concern would be that if cgr.dev is down, any releases or e2e tests of Knative would be broken until it becomes available again, correct?

Yup

If that's something you'd like to explore I can help with that too.

Do you also push to other registries?

[dprotaso]

I'm switching serving over to the chainguard images - my main concern was really GitHub registry stability etc. but with you moving to R2 i'm less concerned now.

/assign @pierDipi for eventing