Document Docker Image Caching changes with v5 release

[tomowatt]

Previous versions of the Plugin, enabled pulling images that were specified via the cache-from parameter. This functionality is no longer available as of v5, due to the inclusion of being able to pass Images in different formats for the cache-from parameter.

Although the breaking change is documented, there is currently no publically available solution to keep this functionality as it was relied upon heavily for it's simplicity. There are workarounds available such as using docker-container driver and using the cache_from & cache_to docker-compose YAML attributes instead but these are not included anywhere in the repository.

This has lead to multiple instances of confusion from users of the Plugin, who upgraded to v5.x and found this loss of functionality and did not have enough information to move forward to get images pulled and to be used as cache.

Example workaround with docker-compose file:

services:
  app:
    build:
    context: ../
    dockerfile: ./Dockerfile
    cache_to:
      - "type=registry,mode=max,image-manifest=true,oci-mediatypes=true,ref={Account ID}.dkr.ecr.us-east-1.amazonaws.com/{Cache Name}:cache"

and using native docker commands to push and use cache using docker-container driver:

$ docker buildx create --name container --driver=docker-container --use --bootstrap
$ docker buildx build --push -t {Account ID}.dkr.ecr.us-east-1.amazonaws.com/{Cache Name}:latest --cache-to type=registry,mode=max,image-manifest=true,oci-mediatypes=true,ref={Account ID}.dkr.ecr.us-east-1.amazonaws.com/{Cache Name}:cache --cache-from type=registry,ref={Account ID}.dkr.ecr.us-east-1.amazonaws.com/{Cache Name}:cache --cache-from type=registry,ref={Account ID}.dkr.ecr.us-east-1.amazonaws.com/{Cache Name}:main .

[tomowatt]

By default, most Docker instances will use the docker build driver, but this by default does not support Cache Exports without changing the Docker Daemon itself via it's config. The docker-container driver is available with most Docker installs and can utilise Cache Exports and pulling from remote as a part of Image Building. So potential, the Plugin could include parameters and/or hooks to enable changing the Build Driver easily for uses.

With this the Plugin currently only has a cache-from parameter and not a cache-to, which should map to cache_to in the docker-compose file. This would allow more abstraction so Cache does not need to be specified from the docker-compose file.

Reference: https://docs.docker.com/build/builders/drivers/

[huewood]

The following code snippets below resolved this issue on our end. For context, we have multiple agents on each CI server so adding unique docker builder containers & appropriate cleanup is required.

pre-command hook:

#!/bin/bash

# Define the builder name
BUILDER_NAME="container-$BUILDKITE_STEP_ID"

# Check if the builder already exists
EXISTING_BUILDER=$(docker buildx ls | grep "$BUILDER_NAME")

if [ -n "$EXISTING_BUILDER" ]; then
    echo "Builder '$BUILDER_NAME' already exists. Removing it..."
    docker buildx rm "$BUILDER_NAME"
fi

# Create the new builder
echo "Creating builder '$BUILDER_NAME'..."
docker buildx create --name "$BUILDER_NAME" --driver docker-container --use --bootstrap

echo "Builder '$BUILDER_NAME' created and set as current."

pre-exit hook (for graceful cleanup in-case of task failure):

#!/bin/bash

# Define the builder name
BUILDER_NAME="container-$BUILDKITE_STEP_ID"

# Check if the builder already exists
EXISTING_BUILDER=$(docker buildx ls | grep "$BUILDER_NAME")

if [ -n "$EXISTING_BUILDER" ]; then
    echo "Builder '$BUILDER_NAME' already exists. Removing it..."
    docker buildx rm "$BUILDER_NAME"
fi 

docker-compose.yml:

 tooling:
    container_name: tooling
    build:
      context: ../..
      dockerfile: ./ci/build/tooling/Dockerfile
      cache_from:
        - type=registry,ref=${DOCKER_REGISTRY}/${IMAGE_REPO}:tooling-cache
      cache_to:
        - type=registry,mode=max,image-manifest=true,oci-mediatypes=true,ref=${DOCKER_REGISTRY}/${IMAGE_REPO}:tooling-cache

I've gone ahead and hardcoded the cache_to & cache_from in the docker-compose.yml but it would be preferable if this is declarable from the plugin inputs as suggested in the original post.