docker backend doesn't use buildx for multi-platform

[mark-thm]

Describe the bug Docker backend runs docker build rather than docker buildx build for multi-platform builds specified with build_platform set to a list. Single valued lists for build_platform will build correctly, but fail to produce a multi-architecture output image.

Pants version 2.18.0a0 (verified bug still present on mainline, source linked below)

OS MacOS

Additional info Docker version 24.0.6, build ed223bc

Using docker build vs docker buildx build: https://github.com/pantsbuild/pants/blob/78111bbc105418dc2f419181e67c8ec2f80f439a/src/python/pants/backend/docker/util_rules/docker_binary.py#L67

Experimenting with docker build directly:

$ docker build --platform=linux/amd64,linux/arm64 .
[+] Building 0.0s (0/0)                                                                                                  docker:desktop-linux
ERROR: Multiple platforms feature is currently not supported for docker driver. Please switch to a different driver (eg. "docker buildx create --use")

[riisi]

@mark-thm Have you tried the following?

1. Enable BuildKit if necessary (it is the default in later versions of Docker):

❯ export DOCKER_BUILDKIT=1

2. Allow Pants to read the env var

[docker]
env_vars = [
  "DOCKER_BUILDKIT",
]

[mark-thm]

Sorry for the delay, just gave that whirl, and docker produces the following stderr output:

ERROR: Multiple platforms feature is currently not supported for docker driver. Please switch to a different driver (eg. "docker buildx create --use")

[mark-thm]

According to the docs BuildKit is default as of Docker 23.0 and I'm Docker 24.0, so I think this has no effect.

[riisi]

Try this (e.g.) also and I think that should get you going for now:

Create a builder using a build driver that is compatible with the cache backend:

❯ docker buildx create --name container --driver=docker-container container

Use the builder:

❯ export BUILDX_BUILDER=container

And also add the BUILDX_BUILDER to the env vars in pants.toml

I came across this as part of this PR.

[mark-thm]

aha, magic. Thanks. I had to drop the final container in the first command to get things to run but that did the trick. Weirdly I'm having to plumb through my PATH to pick up sw_vers from my /usr/bin directory in this setup, but seems like things work. Not sure if this will be well behaved in CI.

[mark-thm]

It seems to me like this level of setup shouldn't be required/I'll have to figure out how to engineer the same set of state onto my CI system vs. just running docker buildx build.

[riisi]

Which CI system are you using? I was just looking into the same for Github Actions recently and it looks straightforward, but not sure about others.

[danny-todd-oxb]

+1 for this issue. Although, I'm having the same error even when pre-creating a builder with buildx following the instructions above and cannot get an image to build.

I'm on macOS 13 Apple Silicon.

[kaos]

the new [docker].use_buildx option from #20154 makes the above a lot easier to tackle, I think. (thanks @riisi!)

[danny-todd-oxb]

@kaos Is there any way to future-pick this feature for an earlier release?

[kaos]

@kaos Is there any way to future-pick this feature for an earlier release?

@danny-todd-oxb It'll be in the 2.19.0 release which has entered stabilization. I don't think it'll come out much sooner if we were to pick it further back, so only reason for us to consider that would be if there are something else blocking you from using the upcoming 2.19 release I think..

[rajeshwar-nu]

I am trying to use docker driver docker-container to build and push multi-platform images.

docker buildx create --name container --driver=docker-container --use --bootstrap
export BUILDX_BUILDER=container # I have set `docker.env_vars=["BUILDX_BUILDER"]` in pants.toml 
pants publish my-target

with

docker_image(
    name="my-target",
    repository="my-repo",
    image_tags=["a-tag"],
    build_platform=["linux/amd64","linux/arm64"],
)

Pants is successfully able to use docker and generate multi-platform images, however it's unable to push, because it's missing the --push flag

WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load

After build is finished, pants tries to push the image manually, however it fails because the image is present in docker-container driver, instead of the expected default docker driver.

I think this can be solved by allowing pants docker subsystem to allow accepting args for docker build command.

Any plans to add support for something like this?

[riisi]

@rajeshwar-nu Yes, this is coming in 2.19 as part of https://github.com/pantsbuild/pants/pull/20154 as mentiond above

[rajeshwar-nu]

@rajeshwar-nu Yes, this is coming in 2.19 as part of #20154 as mentiond above

Sounds Amazing! any ideas when that is coming out?

[chris-smith-zocdoc]

This kinda works now, you can specify the buildx output like this, which causes the image to get pushed to the registry

docker_image(
    build_platform = [
        "linux/amd64",
        "linux/arm64",
    ],
    output={
        'type': 'image,push=true',
    },
)

But it does so under pants package and not pants publish which I don't like. I think a possible solution here would be to have the publish target invoke the same docker buildx build command that package does, but modify the output type from image to image,push=true (or append --push which is equivalent)