New modes to build base-image using existing artifacts

[aroradaman]

What would you like to be added: https://github.com/kubernetes-sigs/kind/pull/3614 allows creating node-images using existing Kubernetes artifacts.

I would like to add a similar option for base-images allowing base-image creation using existing/pre-built aritfacts which includes(containerd, runc, critools, cni-plugins and other artifacts which are built here)

Why is this needed: This will:

1. Speed up base image creation as artifacts can be directly pulled from official releases. For. eg instead of building containerd we can fetch the artifacts from the official containerd releases.
2. Allow testing with custom built artifacts (versions of aritfacts with CGO enabled or versions built with custom patches which are not present in upstream)
3. Allow build/testing in firewalled/airgapped environments where public repositories like https://github.com/opencontainers/containerd can't be accessed.

I understand that kind is primarily designed for testing Kubernetes but this will speed up base-image creation for upstream and might help teams/developers who ship their own Kubernetes versions.

I can take this up if this is acceptable but not a priority due to bandwidth.

[aojea]

@aroradaman can you describe at high level your proposal? the base image on the contrary to the node image , is only a dockerfile

https://github.com/kubernetes-sigs/kind/blob/main/images/base/Dockerfile

[aroradaman]

@aojea Yes, the current base image is a single docker file where we clone upstream repositories and build binaries in different stages. On a high level, one way to achieve the desired customisation is to have multiple stages for preparing binaries (containerd for example) and have build-args control which stage will be called.

ARG CONTAINERD_STAGE=build-containerd

FROM golang:latest AS go-build

# stage for building containerd
# the default behaviour
FROM go-build AS build-containerd
RUN echo "clone and make containerd"
RUN touch /artifact

# stage for pulling containerd artifacts from remote url
FROM go-build AS pull-remote-containerd
RUN echo "download containerd release from remote url"
RUN touch /artifact

# stage for pulling containerd artifacts from local filesystem 
FROM go-build AS pull-containerd-local
RUN echo "copy containerd artifacts from local url"
RUN touch /artifact

# consolitdate containerd_stage
FROM ${CONTAINERD_STAGE} AS containerd_stage

# final stage
FROM go-build AS final
COPY --from=containerd_stage /artifact /artifact

// to build-base image with locally present containerd binaries
$ DOCKER_BUILDKIT=1 docker build -t test . --build-arg CONTAINERD_STAGE=pull-containerd-local

// to build-base image with remote containerd binaries
$ DOCKER_BUILDKIT=1 docker build -t test . --build-arg CONTAINERD_STAGE=pull-remote-containerd

// to build-base image with cloning and building containerd binaries
$ DOCKER_BUILDKIT=1 docker build -t test . --build-arg CONTAINERD_STAGE=build-containerd
OR
$ DOCKER_BUILDKIT=1 docker build -t test .

With build-kit enabled docker will skip the stages which are not required and we can control the flow using build-args. One alternate solution that I can think of is splitting this docker file into multiple docker files, like one for building containerd, one for runc and we can call build only binaries when required and ultimately assemble/copy into the final dockerfile.

[aojea]

why downstream projects can not do that in their own? here there is a control of the sources with the hashes

[aroradaman]

why downstream projects can not do that in their own?

They can, with https://github.com/kubernetes-sigs/kind/pull/3614 I thought there might be scope for doing the same for base-images. Kind itself can benefit from directly pulling the binaries from official release pages rather than cloning the repository and running the build for containerd, runc, cni .. etc, speeding up the base image build process.

[aojea]

This is the diff of #3614 , I'm trying to understand how this will look like for the users , what kind of users will use this and what will be the impact of maintaining such solution ...

One shortcut to use a kubernetes release is to specify the version
directly to pick up the official tar-gzipped files:

kind build node-image v1.30.0

If you prefer to use existing tar-gzipped files like the ones from the kubernetes
release, you can specify those as well from a URL or local directory, for example:

kind build node-image https://dl.k8s.io/v1.30.0/kubernetes-server-linux-arm64.tar.gz kind build node-image $HOME/Downloads/kubernetes-server-linux-amd64.tar.gz

To clear any confusion, you can specify the type of build explicitly
using `--type` parameter, please see the following examples:

[BenTheElder]

I understand that kind is primarily designed for testing Kubernetes but this will speed up base-image creation for upstream [...]

It will not, because we need to keep building the way we do currently, we do it intentionally to ensure that we can control go patching. Not all of the upstreams choose to release new binaries when go releases a new go version.

[...] and might help teams/developers who ship their own Kubernetes versions.

They can already use published base images where the entire base image is pre-built, or they can build teheir own base images.

Kind itself can benefit from directly pulling the binaries from official release pages rather than cloning the repository and running the build for containerd, runc, cni .. etc, speeding up the base image build process.

But we intentionally stopped doing that. Our builds meet our expectations for LICENSING metadata and for quickly patching go CVEs.

The builds are also quite fast. They only take 5 minutes to clone, build and push multi-arch https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/post-kind-push-base-image/1790439773261795328

[BenTheElder]

Allow build/testing in firewalled/airgapped environments where public repositories like https://github.com/opencontainers/containerd can't be accessed.

Unless you're developing kind itself, you should just use pre-built base images.

If you're developing kind itself ... kind itself is on github as are these other projects and your environment should have github access.

Allow testing with custom built artifacts (versions of aritfacts with CGO enabled or versions built with custom patches which are not present in upstream)

You can override the arguments to the image, but we do NOT provide support for base images that do not match upstream, use at your own risk, we're not going to actively encourage that and then eat the support cost.

... and for the binaries that do have CGO enabled, this is again why we don't fetch them from the upstream binaries, because we can't guarantee that the libraries they link to match the base distro we're using (think runc), if you're packaging these binaries into a k8s distro you should be ensuring that they come from a build that matches the linux distro. We do this by building from source in a matching environment because the linux distro packages are very out of date and for testing Kubernetes we require much more current releases of the container ecosystem dependencies.