Cannot set multi-platform bake contexts if platforms do not match exactly

PigeonF commented 1 month ago

Contributing guidelines

[X] I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

[X] ... the documentation does not mention anything about my problem
[X] ... there are no open or closed issues that are related to my problem

Description

When using a target: directive as a bakefile context in a multi-platform scenario, the platforms of the targets have to match exactly or the build fails.

Expected behaviour

The build should succeed if the referencing target is built for a subset of the platforms of the referenced target.

Actual behaviour

The build fails unless the platforms match exactly.

Buildx version

github.com/docker/buildx v0.14.0

Docker info

Client:
 Version:    24.0.9
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.14.0
    Path:     /nix/store/jidnm42865p7pisj8i7nils91ianj19f-docker-plugins/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  2.27.0
    Path:     /nix/store/jidnm42865p7pisj8i7nils91ianj19f-docker-plugins/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 16
 Server Version: 24.0.9
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: journald
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: v1.7.16
 runc version:
 init version:
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.6.32
 Operating System: NixOS 24.05 (Uakari)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 23.37GiB
 Name: geonosis
 ID: 1bde965b-61b0-4cbd-94e0-91a66e0c1109
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  registry.internal
  cache.internal
  127.0.0.0/8
 Registry Mirrors:
  http://cache.internal/
 Live Restore Enabled: true

Builders list

NAME/NODE     DRIVER/ENDPOINT                    STATUS    BUILDKIT               PLATFORMS
remote*       remote
 \_ remote0    \_ tcp://buildkit.internal:3375   running   4cf5e34                linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386
default       docker
 \_ default    \_ default                        running   v0.11.7+435cb77e369c   linux/amd64, linux/amd64/v2, linux/amd64/v3

Configuration

target "default" {
  dockerfile-inline = "FROM other"
  contexts = {
    other = "target:other"
  }
  platforms = [
    "linux/amd64",
    "linux/arm64",
  ]
}

target "other" {
  dockerfile-inline = "FROM docker.io/library/alpine:3"
  platforms = [
    "linux/amd64",
    "linux/arm64",
    "linux/arm/v7",
  ]
}

Build logs

#0 building with "remote" instance using remote driver

#1 [internal] load local bake definitions
#1 reading docker-bake.hcl 314B / 314B done
#1 DONE 0.0s
ERROR: target other can't be used by default because it is defined for different platforms [linux/amd64 linux/arm64 linux/arm/v7] and [linux/amd64 linux/arm64]

Additional info

No response

crazy-max commented 1 month ago

remote*       remote
 \_ remote0    \_ tcp://buildkit.internal:3375   running   4cf5e34                linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386

Seems like your machine does not support arm architectures which is why your build fails with either remote or default builders. You can either append a native arm node to remote builder or setup QEMU on your host as shown in https://docs.docker.com/build/building/multi-platform/#qemu-without-docker-desktop

PigeonF commented 1 month ago

The issue is not that my machine does not support ARM (I can docker buildx bake other fine), just that other cannot be used as the reference. That is, if I can docker buildx bake other I would expect docker buildx bake default to work as well.

You can check this locally by changing the platforms for mid in

https://github.com/docker/buildx/blob/9ad116aa8efabeaf85f62e311508142944408158/bake/bake_test.go#L878-L881

to platforms = ["linux/amd64", "linux/arm64", "linux/arm/v7"]

crazy-max commented 1 month ago

Oh my bad I can't read :see_no_evil:

So yes platforms need to perfectly match otherwise it fails with this error because there is no proper linking between platforms in your named context. Does it work with the following definition?:

target "default" {
  dockerfile-inline = "FROM other"
  contexts = {
    other = "target:other"
  }
  platforms = [
    "linux/amd64",
    "linux/arm64",
    "linux/arm/v7"
  ]
}

target "other" {
  dockerfile-inline = "FROM docker.io/library/alpine:3"
  platforms = [
    "linux/amd64",
    "linux/arm64",
    "linux/arm/v7"
  ]
}

Maybe if a named context does not match platforms of caller, we could override them.

PigeonF commented 1 month ago

Yes, the snippet you link works. The issue is that I want to build two separate targets, one of which supports more platforms than the other 😅 .

I think changing

https://github.com/docker/buildx/blob/9ad116aa8efabeaf85f62e311508142944408158/bake/bake.go#L501-L505

to check if t.Platforms is a subset of t2.Platforms instead of equality should suffice? Or is that what you mean with there not being proper linking between named context and platforms (i.e. this check would pass, but the build would fail down the line because of some platform mismatch)?

crazy-max commented 1 month ago

to check if t.Platforms is a subset of t2.Platforms instead of equality should suffice?

Yeah maybe a subset would be enough but wonder if should not override with platforms from caller instead per my comment https://github.com/docker/buildx/issues/2486#issuecomment-2139598586. WDYT @tonistiigi?

docker / buildx