Add Docker Cache

[wbxyz]

Especially for Debian, where the build-essentials take (~1 minute) to install, a docker cache would help speedup rebuilds.

Reading the docs for docker caching, it seems like there are three relevant options:

• local: A local file system which could potentially be backed by GCS using FUSE
• registry: An OCI registry which could potentially be Artifact Registry
• s3: An S3 bucket (potentially GCS could be used for that?)

I'm currently leaning towards the registry option, using Artifact Registry. The s3 option pointed to a GCS bucket is tempting, because we're already using GCS extensively. However the s3 option is currently in an unreleased state.

The local options is not very appealing because we'd need to map the local file system to some shared resource across all the builds.

• [ ] Choose a registry cache
• [ ] Implement
• [ ] Optimize our builds to maximize usage (place common setup in dedicated steps)

[wbxyz]

Using Artifact registry is pretty straightforward:

docker buildx create --name container-builder --driver docker-container
cat <<'EOS' | docker buildx build --output=type=docker --tag=img --builder=container-builder --cache-to=type=registry,ref=gcr.io/<project>/build-debian:latest --cache-from=type=registry,ref=gcr.io/<project>/build-debian:latest -
{dockerfile}
EOS
docker run --name=container img

Running this on GCB, the cache does help speed up the the build_essentials install in our debian dockerfile, however a lot of this performance increase is lost due to:

• Setuing up the moby/buildkit:buildx-stable-1 container (11 seconds)
• "exporting to docker image format; sending tarball" (42 seconds)
• "importing to docker" (27 seconds)

This is better than the ~120 seconds of apt install build-essentials we're trying to cache, but it's still not ideal.

I suspect this is related to https://github.com/docker/buildx/issues/107. In that issue, it seems the problem might be that we're using docker-container instead of the default docker build driver. We're using docker-container because the default driver doesn't support registry cache backend:

This cache storage backend is not supported with the default docker driver. To use this feature, create a new builder using a different driver.

Frustratingly, a different part of the documentation claims (incorrectly) that the built-in docker driver supports registry:

The default docker driver supports the inline, local, registry, and gha cache backends, but only if you have enabled the containerd image store.

But when attempting to use that we get:

ERROR: cache export feature is currently not supported for docker driver. Please switch to a different driver (eg. "docker buildx create --use")

I experimented by removing --cache-to, changing the --output=type= value (docker, oci, tar) but they all seem to suffer from a long "sending tarball" or have other strange behavior. The best thing I've found so far is to set --output=type=docker,compression=uncompressed but even that still takes 25 seconds for tarball export, which might just be a noisy metric.

For now, I'm going to put a pin in the docker caching. I'd like to run a moderate size benchmark of debian builds to better understand which part of the builds are limiting us (#163 will help there).