platforms `linux/amd64/v2,linux/amd64/v3` not respected, just skipped

the-hotmann commented 12 months ago

Contributing guidelines

[X] I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

[X] ... the documentation does not mention anything about my problem
[X] ... there are no open or closed issues that are related to my problem

Description

I build a container with this script/command:

docker buildx build --pull --push \
    --platform linux/amd64,linux/amd64/v2,linux/amd64/v3,linux/arm64,linux/arm/v4,linux/arm/v5,linux/arm/v6,linux/arm/v7,linux/arm/v8 \
    -t repository/tag .

Dockerfile:

FROM --platform=$BUILDPLATFORM golang:1-alpine as build

ARG TARGETOS TARGETARCH

[...]

RUN GOOS=$TARGETOS GOARCH=$TARGETARCH CGO_ENABLED=0 go build -ldflags "-s -w" -o appName -v main.go

So all the platforms/architectures are getting build, but all these:

linux/amd64 (basically linux/amd64/v1)
linux/amd64/v2
linux/amd64/v3

are 100% the same, and actually they are NOT optimized for the tagged microcode version - they all basically are linux/amd64/linux/amd64/v1. That is good for compatibility, but bad for the optimization, as it basically is just yet another tag that consumes space but does not bring any benefit.

Maybe I am in the wrong here, but as I also build the application for:

linux/arm/v4
linux/arm/v5
linux/arm/v6
linux/arm/v7 (basically linux/arm)
linux/arm/v8

and they all aswell build successfully and are all indeed unique.

If I just build for linux/amd64 and add these linux/amd64/v2 linux/amd64/v3 later, it does not even build them, but instead directly pushes them as they are 'already build' (since it just takes linux/amd64 for it)

Maybe the problem is with me, but I indeed was expecting it to work just like this, as I pass in all the wanted platforms and usually got them out as expected. If the problem is with me, please lead me to the right way - if not, please take this as a bug, or just unexpected behaviour, which does not break anything, but does not work as intended,

Expected behaviour

It really should re-build the app with the new Microarchitecture level and therefore also build an app which is made for the specific Microarchitecture level - not just link/copy them.

Actual behaviour

For ALL linux/amd64 versions it just uses the compiled linux/amd64/v1 build. So you basically always get the same.

Buildx version

github.com/docker/buildx v0.11.2 9872040

Docker info

Client: Docker Engine - Community
 Version:    24.0.7
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.21.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 7
  Running: 7
  Paused: 0
  Stopped: 0
 Images: 8
 Server Version: 24.0.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 runc version: v1.1.9-0-gccaecfc
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.10.0-26-amd64
 Operating System: Debian GNU/Linux 11 (bullseye)
 OSType: linux
 Architecture: x86_64
 CPUs: 10
 Total Memory: 31.35GiB
 Name: clicon.hosting-hotmann.de
 ID: ZUV6:WTSH:O2CJ:ZU5E:OPCZ:2K6B:X4BB:BAQS:LY23:ADAG:JPIL:EOHT
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: h0tmann
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Builders list

NAME/NODE    DRIVER/ENDPOINT             STATUS  BUILDKIT             PLATFORMS
multiarch *  docker-container                                         
  multiarch0 unix:///var/run/docker.sock running v0.11.6              linux/amd64*, linux/amd64/v2*, linux/amd64/v3*, linux/386*, linux/arm64*, linux/arm/v8*, linux/arm/v7*, linux/arm/v6*, linux/riscv64*, linux/ppc64le*, linux/s390x*, linux/ppc64, linux/mips64le, linux/mips64
default      docker                                                   
  default    default                     running v0.11.7+d3e6c1360f6e linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/arm64, linux/riscv64, linux/ppc64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6

Configuration

FROM --platform=$BUILDPLATFORM golang:1-alpine as build

ARG TARGETOS TARGETARCH

[...]

RUN GOOS=$TARGETOS GOARCH=$TARGETARCH CGO_ENABLED=0 go build -ldflags "-s -w" -o appName -v main.go

Build logs

No response

Additional info

For linux/arm the microarchitecture level gets passed on, which I also can specify, but when I do the same for linux/amd64 it does not behave the same.

tonistiigi commented 12 months ago

When building Go applications, you would use GOAMD64 environment variable to configure the compiler. https://github.com/golang/go/wiki/MinimumRequirements#microarchitecture-support

ARG TARGETVARIANT
RUN GOAMD64=$TARGETVARIANT GOOS=...

If you get different binaries or the same depends on if your app uses some feature where Go compiler produces different machine code.

the-hotmann commented 12 months ago

Yeah I know this, I usually work around this by using this RUN command:

RUN if [ "$TARGETARCH" == "AMD64" ] && [ "$TARGETVARIANT" != "" ]; then \
        GOOS=$TARGETOS GOARCH=$TARGETARCH GOAMD64=$TARGETVARIANT CGO_ENABLED=0 go build -ldflags "-s -w" -o appName -v main.go; \
    else \
        GOOS=$TARGETOS GOARCH=$TARGETARCH CGO_ENABLED=0 go build -ldflags "-s -w" -o appName -v main.go; \
    fi

But I just dont get, why it is just working fine for linux/arm/v4, linux/arm/v5, linux/arm/v6, linux/arm/v7 even if I do NOT use $TARGETVARIANT at all!?

For me this sounds like this is unexpected behaviour.

tonistiigi commented 12 months ago

It is probably the base image that is different for your arm builds, not the generated code from compiler. In Go you would still need to pass GOARM same way to control the generated code. It isn't a direct GOARM=$TARGETVARIANT setting there though so helper scripts like xx https://github.com/tonistiigi/xx/blob/master/src/xx-go#L12 are needed to do it without lot of if statements in the Dockerfile.

tonistiigi commented 12 months ago

if [ "$TARGETARCH" == "AMD64" ]

Btw, these comparisons are case-sensitive so this would never match.

the-hotmann commented 12 months ago

Btw, these comparisons are case-sensitive so this would never match.

My mistake, this actually did the trick - thank you! GOAMD64 thats why I mixed it up. Still I think that this is somehow weird. Should be enough to pass the plattforms, but golang does not support this as simple as this.

docker / buildx