search

docker-library / busybox

Docker Official Image packaging for Busybox
http://busybox.net
388 stars 126 forks source link

Manifest issues for 7 architectures on versions 1.35 and 1.36 #189

Closed lincolnthalles closed 6 months ago

lincolnthalles commented 7 months ago

Only 386 and amd64 have consistent manifests and works in all cases, and for all image versions.

Builds using Busybox 1.35 and 1.36 are consistently failing for:

The following Dockerfile only succeeds for 386 and amd64:

FROM --platform=${BUILDPLATFORM:-linux/amd64} i386/busybox:1.36 as target-386
FROM --platform=${BUILDPLATFORM:-linux/amd64} amd64/busybox:1.36 as target-amd64
FROM --platform=${BUILDPLATFORM:-linux/amd64} arm32v5/busybox:1.36 as target-arm32v5
FROM --platform=${BUILDPLATFORM:-linux/amd64} arm32v6/busybox:1.36 as target-arm32v6
FROM --platform=${BUILDPLATFORM:-linux/amd64} arm32v7/busybox:1.36 as target-arm32v7
FROM --platform=${BUILDPLATFORM:-linux/amd64} arm64v8/busybox:1.36 as target-arm64
FROM --platform=${BUILDPLATFORM:-linux/amd64} ppc64le/busybox:1.36 as target-ppc64le
FROM --platform=${BUILDPLATFORM:-linux/amd64} riscv64/busybox:1.36 as target-riscv64
FROM --platform=${BUILDPLATFORM:-linux/amd64} s390x/busybox:1.36 as target-s390x

By removing --platform=${BUILDPLATFORM:-linux/amd64}, 1.35 and 1.36 build succeeds for 386, amd64, arm64, ppc64le, riscv64, s390x, though arm32v5, arm32v6 and arm32v7 still fail:

FROM i386/busybox:1.36 as target-386
FROM amd64/busybox:1.36 as target-amd64
FROM arm32v5/busybox:1.36 as target-arm32v5
FROM arm32v6/busybox:1.36 as target-arm32v6
FROM arm32v7/busybox:1.36 as target-arm32v7
FROM arm64v8/busybox:1.36 as target-arm64
FROM ppc64le/busybox:1.36 as target-ppc64le
FROM riscv64/busybox:1.36 as target-riscv64
FROM s390x/busybox:1.36 as target-s390x

Version 1.34 succeeds for all architectures, with and without --platform:

FROM --platform=${BUILDPLATFORM:-linux/amd64} i386/busybox:1.34 as target-386
FROM --platform=${BUILDPLATFORM:-linux/amd64} amd64/busybox:1.34 as target-amd64
FROM --platform=${BUILDPLATFORM:-linux/amd64} arm32v5/busybox:1.34 as target-arm32v5
FROM --platform=${BUILDPLATFORM:-linux/amd64} arm32v6/busybox:1.34 as target-arm32v6
FROM --platform=${BUILDPLATFORM:-linux/amd64} arm32v7/busybox:1.34 as target-arm32v7
FROM --platform=${BUILDPLATFORM:-linux/amd64} arm64v8/busybox:1.34 as target-arm64
FROM --platform=${BUILDPLATFORM:-linux/amd64} ppc64le/busybox:1.34 as target-ppc64le
FROM --platform=${BUILDPLATFORM:-linux/amd64} riscv64/busybox:1.34 as target-riscv64
FROM --platform=${BUILDPLATFORM:-linux/amd64} s390x/busybox:1.34 as target-s390x
# 1.35
=> ERROR [internal] load metadata for docker.io/arm32v5/busybox:1.35
ERROR: failed to solve: arm32v5/busybox:1.35: no match for platform in manifest sha256:f85be61f1161865b122af164b7ed4c6332205dd9d3c78ec558e1f0975083a9f1: not found

=> ERROR [internal] load metadata for docker.io/arm32v6/busybox:1.35
ERROR: failed to solve: arm32v6/busybox:1.35: no match for platform in manifest sha256:48fee3c3b3740afe88accb7221e38e7bc53444f3107fa0b0cba042c0d84b824e: not found

=> ERROR [internal] load metadata for docker.io/arm32v7/busybox:1.35
ERROR: failed to solve: arm32v7/busybox:1.35: no match for platform in manifest sha256:a08e8b1ac2a5fa03cd445bf53dbf782aae8be654629584bd5e7fb9dd0da9f6fc: not found

=> ERROR [internal] load metadata for docker.io/arm64v8/busybox:1.35
ERROR: failed to solve: arm64v8/busybox:1.35: no match for platform in manifest sha256:e9a6faa98aa959c3648c29bef33b47e4432d8909100edfe149e06abf12db40dc: not found

=> ERROR [internal] load metadata for docker.io/riscv64/busybox:1.35
ERROR: failed to solve: riscv64/busybox:1.35: no match for platform in manifest sha256:32756a1a756daaa3263a0f5260f9c160c9c9b9213a2b01e9dccfcfc152ab8e87: not found

=> ERROR [internal] load metadata for docker.io/ppc64le/busybox:1.35
ERROR: failed to solve: ppc64le/busybox:1.35: no match for platform in manifest sha256:8b66cb63e44fcfdd985047af26ecf00a123c9699538fc43e6acc53edf3568ba5: not found

=> ERROR [internal] load metadata for docker.io/s390x/busybox:1.35
ERROR: failed to solve: s390x/busybox:1.35: no match for platform in manifest sha256:cc6afefbda518a30d457708512710cac7db77d4761d79d72344034e1e85665a4: not found

# 1.36
=> ERROR [internal] load metadata for docker.io/arm32v5/busybox:1.36
ERROR: failed to solve: arm32v5/busybox:1.36: no match for platform in manifest sha256:e521011ef2d871178fad87305c6c6b05b873714516af0caf8e43e4c302264244: not found

=> ERROR [internal] load metadata for docker.io/arm32v6/busybox:1.36
ERROR: failed to solve: arm32v6/busybox:1.36: no match for platform in manifest sha256:7149179e1d86dbbd5d4ade2cb356dc7f1c80dd3639ee400421b4864abbe58cba: not found

=> ERROR [internal] load metadata for docker.io/arm32v7/busybox:1.36
ERROR: failed to solve: arm32v7/busybox:1.36: no match for platform in manifest sha256:4d5d73a7dad92c3bcc32aee7c033f05073436afc6826f4d3b3072ee2d02fc18d: not found

=> ERROR [internal] load metadata for docker.io/arm64v8/busybox:1.36
ERROR: failed to solve: arm64v8/busybox:1.36: no match for platform in manifest sha256:8577c15f745026b8570a81f48bf4d8c95f4cd56d8f9842880c18cd4962b59458: not found

=> ERROR [internal] load metadata for docker.io/riscv64/busybox:1.36
ERROR: failed to solve: riscv64/busybox:1.36: no match for platform in manifest sha256:9175380f4d5888df9ec118e954423120598d45e1310e53385f32edc7d76d716e: not found

=> ERROR [internal] load metadata for docker.io/ppc64le/busybox:1.36
ERROR: failed to solve: ppc64le/busybox:1.36: no match for platform in manifest sha256:8e76c99afe028310998d516d39d53b0d6cc5e347a1ece4834bbaf8322e2eafd0: not found

=> ERROR [internal] load metadata for docker.io/s390x/busybox:1.36
ERROR: failed to solve: s390x/busybox:1.36: no match for platform in manifest sha256:91eb9ed96ddf1bd08d329ce13543de75e410862df9b9f8242ea5be07534e8a5b: not found

This mixed syntax gets the most recent images I'm able to use:

FROM --platform=${BUILDPLATFORM:-linux/amd64} i386/busybox:1.36 as target-386
FROM --platform=${BUILDPLATFORM:-linux/amd64} amd64/busybox:1.36 as target-amd64
FROM --platform=${BUILDPLATFORM:-linux/amd64} arm32v5/busybox:1.34 as target-arm32v5
FROM --platform=${BUILDPLATFORM:-linux/amd64} arm32v6/busybox:1.34 as target-arm32v6
FROM --platform=${BUILDPLATFORM:-linux/amd64} arm32v7/busybox:1.34 as target-arm32v7
FROM arm64v8/busybox:1.36 as target-arm64
FROM ppc64le/busybox:1.36 as target-ppc64le
FROM riscv64/busybox:1.36 as target-riscv64
FROM s390x/busybox:1.36 as target-s390x

This is a multi-stage cross-platform build, and this syntax works fine for Debian and Alpine (with the exception of arm32v5, which Alpine doesn't support). I also checked each built image with uname -m and it shows the expected architecture.

tianon commented 7 months ago

Indeed, this is a quirk of some of our build updates -- we can't provide provenance / attestation data without switching from a raw image manifest to an image index, which means they have proper platform specifications and thus the --platform needs to be specified correctly (even on the per-arch repositories, where it wasn't previously necessary except for Windows-based images).

I'm not sure what your intent with so many FROM values is -- can you elaborate a bit on the use case? Why are they all effectively --platform=$BUILDPLATFORM? Why not simply FROM busybox with an appropriate --platform flag instead (FROM --platform=linux/arm/v7 busybox:1.36 for example)?

lincolnthalles commented 6 months ago

Indeed, this is a quirk of some of our build updates -- we can't provide provenance / attestation data without switching from a raw image manifest to an image index, which means they have proper platform specifications and thus the --platform needs to be specified correctly (even on the per-arch repositories, where it wasn't previously necessary except for Windows-based images).

I'm not sure what your intent with so many FROM values is -- can you elaborate a bit on the use case? Why are they all effectively --platform=$BUILDPLATFORM? Why not simply FROM busybox with an appropriate --platform flag instead (FROM --platform=linux/arm/v7 busybox:1.36 for example)?

I was trying to achieve cross-compilation with --platform=$BUILDPLATFORM.

Some distro images, and even architecture-targeted repositories are using arm32, and newer Busybox images are tagged with just "arm".

Looks like a big part of the issue is the tags for arm32/v5-v7 are mismatched between image versions and Docker doesn't do any sort of normalization or tries to fallback to aliases. And to make things worse, users can tag platforms in different ways (arm and arm32, for instance).

I was able to get newer Busybox arm32 images using FROM --platform=linux/arm/v6 busybox:1.36 as target-arm32v6. This compatibilizes Busybox's current platform layout with mine.

I figured it's also possible to remove --platform entirely, or even use a single FROM busybox:1.36 by changing buildx to use exactly the same platform layout current Busybox images are using, but using multiple FROM's allows changing some images to Alpine, for example.

This is a working sample (calling buildx with --platform=linux/arm32v5 and derivations):

FROM --platform=${BUILDPLATFORM:-linux/amd64} gcr.io/distroless/static:latest as distroless
FROM busybox:1.36 as target-386
FROM busybox:1.36 as target-amd64
FROM --platform=linux/arm/v5 busybox:1.36 as target-arm32v5
FROM --platform=linux/arm/v6 busybox:1.36 as target-arm32v6
FROM --platform=linux/arm/v7 busybox:1.36 as target-arm32v7
FROM busybox:1.36 as target-arm64
FROM busybox:1.36 as target-ppc64le
FROM busybox:1.36 as target-riscv64
FROM busybox:1.36 as target-s390x

# Select appropriate base image.
FROM target-${TARGETARCH}${TARGETVARIANT}

COPY --from=distroless /usr/share/zoneinfo /usr/share/zoneinfo
COPY --from=distroless /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/

# Copy proper application binary.
COPY build/backend/linux_${TARGETARCH}${TARGETVARIANT}/app /opt/app/

It's not consistent, but it's working.

Anyway, I appreciate your help, and thanks for your time.