search

docker / buildx

Docker CLI plugin for extended build capabilities with BuildKit
Apache License 2.0
3.54k stars 481 forks source link

Random "SIGSEGV: segmentation violation code" when using "docker buildx imagetools create" #1768

Open mbentley opened 1 year ago

mbentley commented 1 year ago

Contributing guidelines

I've found a bug and checked that ...

Description

When running a docker buildx imagetools create... command in my CI, it will occasionally panic on a random repo. It doesn't seem to be specific to any particular images/builds.

Expected behaviour

It wouldn't panic. Seems to be different than issues like https://github.com/docker/buildx/issues/1521, https://github.com/docker/buildx/issues/1425

Actual behaviour

$ docker buildx imagetools create --progress plain -t mbentley/alpine:3.12 mbentley/alpine:3.12-amd64 mbentley/alpine:3.12-arm64 mbentley/alpine:3.12-armv7
 panic: runtime error: invalid memory address or nil pointer dereference
 [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x109259e]

 goroutine 28 [running]:
 go.opentelemetry.io/contrib/instrumentation/net/http/httptrace/otelhttptrace.(*clientTracer).end(0xc000639080, {0x201d4cd, 0xc}, {0x0?, 0x0?}, {0xc00054bf00?, 0x4, 0x4})
    go.opentelemetry.io/contrib/instrumentation/net/http/httptrace/otelhttptrace@v0.29.0/clienttrace.go:231 +0x77e
 go.opentelemetry.io/contrib/instrumentation/net/http/httptrace/otelhttptrace.(*clientTracer).gotConn(0x237da28?, {{0x2385c78?, 0xc00048ee00?}, 0x28?, 0x7d?, 0xc0005d01a8?})
    go.opentelemetry.io/contrib/instrumentation/net/http/httptrace/otelhttptrace@v0.29.0/clienttrace.go:288 +0x64c
 net/http.http2traceGotConn(0xc00043a1a0?, 0xc0004c0a80, 0x0)
    net/http/h2_bundle.go:9778 +0x203
 net/http.(*http2Transport).RoundTripOpt(0xc000144200, 0xc00054bd00, {0x40?})
    net/http/h2_bundle.go:7248 +0x1a9
 net/http.(*http2Transport).RoundTrip(...)
    net/http/h2_bundle.go:7210
 net/http.http2noDialH2RoundTripper.RoundTrip({0x32eff40?}, 0xc00054bd00?)
    net/http/h2_bundle.go:9742 +0x1b
 net/http.(*Transport).roundTrip(0x32eff40, 0xc00054bd00)
    net/http/transport.go:540 +0x38a
 net/http.(*Transport).RoundTrip(0xc00054bd00?, 0x237da28?)
    net/http/roundtrip.go:17 +0x19
 go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*Transport).RoundTrip(0xc000269ab0, 0xc00054bc00)
    go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.29.0/transport.go:116 +0x5ba
 net/http.send(0xc00054bc00, {0x2360ae0, 0xc000269ab0}, {0x1facca0?, 0xc0005d0a01?, 0x0?})
    net/http/client.go:251 +0x5f7
 net/http.(*Client).send(0xc000b0cde0, 0xc00054bc00, {0x7f158409fc80?, 0xc00040c670?, 0x0?})
    net/http/client.go:175 +0x9b
 net/http.(*Client).do(0xc000b0cde0, 0xc00054bc00)
    net/http/client.go:715 +0x8fc
 net/http.(*Client).Do(...)
    net/http/client.go:581
 golang.org/x/net/context/ctxhttp.Do({0x237da28?, 0xc000b0cd50}, 0xc000b0cd50?, 0xc000498600)
    golang.org/x/net@v0.4.0/context/ctxhttp/ctxhttp.go:27 +0xf2
 github.com/containerd/containerd/remotes/docker.(*request).do(0xc0005b0510, {0x237da28, 0xc000b0cbd0})
    github.com/containerd/containerd@v1.6.16-0.20230124210447-1709cfe273d9/remotes/docker/resolver.go:570 +0x5ac
 github.com/containerd/containerd/remotes/docker.(*request).doWithRetries(0x1e856c0?, {0x237da28, 0xc000b0cbd0}, {0x0, 0x0, 0x0})
    github.com/containerd/containerd@v1.6.16-0.20230124210447-1709cfe273d9/remotes/docker/resolver.go:579 +0x4a
 github.com/containerd/containerd/remotes/docker.dockerFetcher.open({0xc0005b0510?}, {0x237da28, 0xc000b0cbd0}, 0xc0005b0510, {0xc00005cc30?, 0x201161e?}, 0x0)
    github.com/containerd/containerd@v1.6.16-0.20230124210447-1709cfe273d9/remotes/docker/fetcher.go:164 +0x305
 github.com/containerd/containerd/remotes/docker.dockerFetcher.Fetch.func1(0x203000?)
    github.com/containerd/containerd@v1.6.16-0.20230124210447-1709cfe273d9/remotes/docker/fetcher.go:131 +0x510
 github.com/containerd/containerd/remotes/docker.(*httpReadSeeker).reader(0xc000756400)
    github.com/containerd/containerd@v1.6.16-0.20230124210447-1709cfe273d9/remotes/docker/httpreadseeker.go:146 +0xd5
 github.com/containerd/containerd/remotes/docker.(*httpReadSeeker).Read(0xc000756400, {0xc000b78000, 0x200, 0x200})
    github.com/containerd/containerd@v1.6.16-0.20230124210447-1709cfe273d9/remotes/docker/httpreadseeker.go:52 +0x65
 bytes.(*Buffer).ReadFrom(0xc000b0cc00, {0x7f15843ae058, 0xc000756400})
    bytes/buffer.go:202 +0x98
 io.copyBuffer({0x235de60, 0xc000b0cc00}, {0x7f15843ae058, 0xc000756400}, {0x0, 0x0, 0x0})
    io/io.go:413 +0x14b
 io.Copy(...)
    io/io.go:386
 github.com/docker/buildx/util/imagetools.(*Resolver).GetDescriptor(0x24?, {0x237d980, 0xc000af4500}, {0xc00005cba0, 0x24}, {{0xc00005cc30, 0x2e}, {0xc000164280, 0x47}, 0x35c, ...})
    github.com/docker/buildx/util/imagetools/inspect.go:109 +0x137
 github.com/docker/buildx/util/imagetools.(*Resolver).loadPlatform(0xc00063c0c0?, {0x237d980, 0xc000af4500}, 0xc000300180, {0xc00005cba0, 0x24}, {0xc00055e000, 0x1f6, 0x200})
    github.com/docker/buildx/util/imagetools/create.go:216 +0x1a5
 github.com/docker/buildx/util/imagetools.(*Resolver).Combine.func1.1()
    github.com/docker/buildx/util/imagetools/create.go:59 +0x35d
 golang.org/x/sync/errgroup.(*Group).Go.func1()
    golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 +0x64
 created by golang.org/x/sync/errgroup.(*Group).Go
    golang.org/x/sync@v0.1.0/errgroup/errgroup.go:72 +0xa5

Buildx version

github.com/docker/buildx v0.10.4 c513d34049e499c53468deac6c4267ee72948f02

Docker info

$ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.4
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  pushrm: Push Readme to container registry (Christian Korneck)
    Version:  1.9.0
    Path:     /var/lib/jenkins/.docker/cli-plugins/docker-pushrm

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 18
 Server Version: 23.0.5
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: local
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2806fc1057397dbaeefbea0e4e17bddfbd388f38
 runc version: v1.1.5-0-gf19387a
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.10.0-21-amd64
 Operating System: Ubuntu 20.04.6 LTS (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 125.5GiB
 Name: builder1
 ID: OBX6:FGPU:MEEZ:42R4:OQUL:C2D2:2X7Z:2KRF:HAPK:STUW:LLW4:P4OB
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: mbentley
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Registry Mirrors:
  https://registry-mirror.casa.mbentley.net/
 Live Restore Enabled: false

Builders list

NAME/NODE             DRIVER/ENDPOINT          STATUS  BUILDKIT PLATFORMS
arm64-buildkit-test   remote
  arm64-buildkit-test tcp://192.168.5.3:1235   running b0c05cd  linux/arm64*, linux/arm/v7*, linux/arm/v6*
arm64-buildkit1       remote
  arm64-buildkit1     tcp://192.168.5.3:1234   running b0c05cd  linux/arm64*, linux/arm/v7*, linux/arm/v6*
builder1 *            remote
  buildkitlb          tcp://192.168.2.165:1234 running b0c05cd  linux/amd64*, linux/arm64, linux/arm/v7, linux/arm/v6, linux/amd64/v2, linux/amd64/v3, linux/amd64/v4, linux/riscv64, linux/ppc64, linux/ppc64le, linux/s390x, linux/mips64le, linux/mips64
  arm64-buildkit1     tcp://192.168.5.3:1234   running b0c05cd  linux/arm64*, linux/arm/v7*, linux/arm/v6*
buildkit-test-lb      remote
  buildkittestlb      tcp://192.168.2.165:1235 running b0c05cd  linux/amd64*, linux/arm64, linux/arm/v7, linux/arm/v6, linux/amd64/v2, linux/amd64/v3, linux/amd64/v4, linux/riscv64, linux/ppc64, linux/ppc64le, linux/s390x, linux/mips64le, linux/mips64
  arm64-buildkit-test tcp://192.168.5.3:1235   running b0c05cd  linux/arm64*, linux/arm/v7*, linux/arm/v6*
buildkit-test1        remote
  buildkit-test1      tcp://192.168.2.171:1234 running b0c05cd  linux/amd64*, linux/arm64*, linux/arm/v7*, linux/arm/v6*, linux/amd64/v2, linux/amd64/v3, linux/amd64/v4, linux/riscv64, linux/ppc64, linux/ppc64le, linux/s390x, linux/mips64le, linux/mips64
buildkit-test2        remote
  buildkit-test2      tcp://192.168.2.172:1234 running b0c05cd  linux/amd64*, linux/arm64*, linux/arm/v7*, linux/arm/v6*, linux/amd64/v2, linux/amd64/v3, linux/amd64/v4, linux/riscv64, linux/ppc64, linux/ppc64le, linux/s390x, linux/mips64le, linux/mips64
buildkit1             remote
  buildkit1           tcp://192.168.2.168:1234 running b0c05cd  linux/amd64*, linux/arm64*, linux/arm/v7*, linux/arm/v6*, linux/amd64/v2, linux/amd64/v3, linux/amd64/v4, linux/riscv64, linux/ppc64, linux/ppc64le, linux/s390x, linux/mips64le, linux/mips64
buildkit2             remote
  buildkit2           tcp://192.168.2.169:1234 running b0c05cd  linux/amd64*, linux/arm64*, linux/arm/v7*, linux/arm/v6*, linux/amd64/v2, linux/amd64/v3, linux/amd64/v4, linux/riscv64, linux/ppc64, linux/ppc64le, linux/s390x, linux/mips64le, linux/mips64
buildkit3             remote
  buildkit3           tcp://192.168.2.170:1234 running b0c05cd  linux/amd64*, linux/arm64*, linux/arm/v7*, linux/arm/v6*, linux/amd64/v2, linux/amd64/v3, linux/amd64/v4, linux/riscv64, linux/ppc64, linux/ppc64le, linux/s390x, linux/mips64le, linux/mips64
default               docker
  default             default                  running 23.0.5   linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/amd64/v4, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6

Configuration

Here is the Dockerfile from the build:

# STAGE 0: use base image from Docker Hub and upgrade the existing packages
ARG IMAGE_TAG="${IMAGE_TAG:-3.17}"
FROM alpine:${IMAGE_TAG} AS base

RUN apk --no-cache upgrade --purge

# STAGE 1: copy contents of the original base image to a new image so we don't have overlapping files in layers
FROM scratch
COPY --from=base / /
LABEL maintainer="Matt Bentley <mbentley@mbentley.net>"
CMD ["/bin/sh"]

Then the build commands:

docker buildx build --output type=registry,buildinfo=false --provenance=false --builder ${NODE_NAME} --pull --push --progress plain --no-cache=false --build-arg IMAGE_TAG="3.12" --platform linux/amd64 -t mbentley/alpine:3.12-amd64 -f Dockerfile --cache-from=type=registry,ref=registry.casa.mbentley.net/mbentley/alpine:3.12-amd64-cache --cache-to=type=registry,ref=registry.casa.mbentley.net/mbentley/alpine:3.12-amd64-cache,mode=min .
docker buildx build --output type=registry,buildinfo=false --provenance=false --builder ${NODE_NAME} --pull --push --progress plain --no-cache=false --build-arg IMAGE_TAG="3.12" --platform linux/arm64 -t mbentley/alpine:3.12-arm64 -f Dockerfile --cache-from=type=registry,ref=registry.casa.mbentley.net/mbentley/alpine:3.12-arm64-cache --cache-to=type=registry,ref=registry.casa.mbentley.net/mbentley/alpine:3.12-arm64-cache,mode=min .
docker buildx build --output type=registry,buildinfo=false --provenance=false --builder ${NODE_NAME} --pull --push --progress plain --no-cache=false --build-arg IMAGE_TAG="3.12" --platform linux/arm/v7 -t mbentley/alpine:3.12-armv7 -f Dockerfile --cache-from=type=registry,ref=registry.casa.mbentley.net/mbentley/alpine:3.12-armv7-cache --cache-to=type=registry,ref=registry.casa.mbentley.net/mbentley/alpine:3.12-armv7-cache,mode=min .

And then the docker buildx imagetools create... command:

docker buildx imagetools create --progress plain -t mbentley/alpine:3.12 mbentley/alpine:3.12-amd64 mbentley/alpine:3.12-arm64 mbentley/alpine:3.12-armv7

Build logs

n/a

Additional info

No response

jensh007 commented 1 year ago

I ran into the same issue even with the example given in the documentation.

docker buildx imagetools inspect --raw alpine | jq '.manifests[0] | .platform."os.version"="10.1"' > descr.json
$ docker buildx imagetools create -f descr.json myuser/image

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x20 pc=0x1051a3e1c]

goroutine 23 [running]:
github.com/docker/buildx/util/imagetools.(*Resolver).Combine.func1.1()
    /src/util/imagetools/create.go:35 +0x6c
golang.org/x/sync/errgroup.(*Group).Go.func1()
    /src/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x60
created by golang.org/x/sync/errgroup.(*Group).Go
    /src/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0xa8
~/SAPDevelop/git/ocm/ocmdockerbuild> docker buildx imagetools create -f alpine.json -t mypatched:1.0.0
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x20 pc=0x101667e1c]

goroutine 39 [running]:
github.com/docker/buildx/util/imagetools.(*Resolver).Combine.func1.1()
    /src/util/imagetools/create.go:35 +0x6c
golang.org/x/sync/errgroup.(*Group).Go.func1()
    /src/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x60
created by golang.org/x/sync/errgroup.(*Group).Go
    /src/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0xa8

Is there any ETA when this gets fixed?

crazy-max commented 1 year ago

github.com/docker/buildx v0.10.4 c513d34

Do you repro also with latest stable v0.11.0?

mbentley commented 1 year ago

Yes I can. I am now running github.com/docker/buildx v0.11.0 687feca9e8dcd1534ac4c026bc4db5a49de0dd6e

I had another failure two days ago:

+ docker buildx imagetools create --progress plain -t mbentley/omada-controller:5.6 mbentley/omada-controller:5.6-amd64 mbentley/omada-controller:5.6-arm64 mbentley/omada-controller:5.6-armv7l
 panic: runtime error: invalid memory address or nil pointer dereference
 [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x117206e]

 goroutine 30 [running]:
 go.opentelemetry.io/contrib/instrumentation/net/http/httptrace/otelhttptrace.(*clientTracer).end(0xc000a40180, {0x236d094, 0xc}, {0x0?, 0x0?}, {0xc0000e8b00?, 0x4, 0x4})
    go.opentelemetry.io/contrib/instrumentation/net/http/httptrace/otelhttptrace@v0.40.0/clienttrace.go:231 +0x76e
 go.opentelemetry.io/contrib/instrumentation/net/http/httptrace/otelhttptrace.(*clientTracer).gotConn(0x26de9a8?, {{0x26e6f98?, 0xc0005fc000?}, 0xf8?, 0x5e?, 0x1?})
    go.opentelemetry.io/contrib/instrumentation/net/http/httptrace/otelhttptrace@v0.40.0/clienttrace.go:288 +0x645
 net/http.http2traceGotConn(0xc00071c000?, 0xc000002c00, 0x0)
    net/http/h2_bundle.go:10096 +0x1ee
 net/http.(*http2Transport).RoundTripOpt(0xc0001b5b90, 0xc0000e8900, {0xa0?})
    net/http/h2_bundle.go:7522 +0x1ac
 net/http.(*http2Transport).RoundTrip(...)
    net/http/h2_bundle.go:7475
 net/http.http2noDialH2RoundTripper.RoundTrip({0x37dc480?}, 0xc0000e8900?)
    net/http/h2_bundle.go:10060 +0x1b
 net/http.(*Transport).roundTrip(0x37dc480, 0xc0000e8900)
    net/http/transport.go:548 +0x3ca
 net/http.(*Transport).RoundTrip(0x37dbbc0?, 0x26de9a8?)
    net/http/roundtrip.go:17 +0x19
 go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*Transport).RoundTrip(0xc00029d500, 0xc0000e8100)
    go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.40.0/transport.go:116 +0x5e2
 net/http.send(0xc0000e8100, {0x26bf500, 0xc00029d500}, {0x10?, 0x22f16a0?, 0x0?})
    net/http/client.go:252 +0x5f7
 net/http.(*Client).send(0xc0005f2660, 0xc0000e8100, {0x7f9af591a8f0?, 0xc0006406c0?, 0x0?})
    net/http/client.go:176 +0x9b
 net/http.(*Client).do(0xc0005f2660, 0xc0000e8100)
    net/http/client.go:716 +0x8fb
 net/http.(*Client).Do(...)
    net/http/client.go:582
 github.com/containerd/containerd/remotes/docker.(*request).do(0xc0005f6090, {0x26de9a8, 0xc0005f2480})
    github.com/containerd/containerd@v1.7.2/remotes/docker/resolver.go:589 +0x686
 github.com/containerd/containerd/remotes/docker.(*request).doWithRetries(0x21a16a0?, {0x26de9a8, 0xc0005f2480}, {0x0, 0x0, 0x0})
    github.com/containerd/containerd@v1.7.2/remotes/docker/resolver.go:600 +0x4a
 github.com/containerd/containerd/remotes/docker.dockerFetcher.open({0xc0005f6090?}, {0x26de9a8, 0xc0005f2480}, 0xc0005f6090, {0xc0005b0960?, 0x2360d3e?}, 0x0)
    github.com/containerd/containerd@v1.7.2/remotes/docker/fetcher.go:262 +0x3d7
 github.com/containerd/containerd/remotes/docker.dockerFetcher.Fetch.func1(0x40dbea?)
    github.com/containerd/containerd@v1.7.2/remotes/docker/fetcher.go:131 +0x8cc
 github.com/containerd/containerd/remotes/docker.(*httpReadSeeker).reader(0xc000126c00)
    github.com/containerd/containerd@v1.7.2/remotes/docker/httpreadseeker.go:146 +0xb8
 github.com/containerd/containerd/remotes/docker.(*httpReadSeeker).Read(0xc000126c00, {0xc0005ec200, 0x200, 0x200})
    github.com/containerd/containerd@v1.7.2/remotes/docker/httpreadseeker.go:52 +0x45
 bytes.(*Buffer).ReadFrom(0xc0005f24b0, {0x7f9af5506098, 0xc000126c00})
    bytes/buffer.go:202 +0x98
 io.copyBuffer({0x26bb420, 0xc0005f24b0}, {0x7f9af5506098, 0xc000126c00}, {0x0, 0x0, 0x0})
    io/io.go:413 +0x14b
 io.Copy(...)
    io/io.go:386
 github.com/docker/buildx/util/imagetools.(*Resolver).GetDescriptor(0x9?, {0x26de900, 0xc00013a7d0}, {0xc0005b0930, 0x2d}, {{0xc0005b0960, 0x2e}, {0xc00073c3c0, 0x47}, 0x11aa, ...})
    github.com/docker/buildx/util/imagetools/inspect.go:109 +0x137
 github.com/docker/buildx/util/imagetools.(*Resolver).loadPlatform(0xc000551700?, {0x26de900, 0xc00013a7d0}, 0xc0005db3e0, {0xc0005b0930, 0x2d}, {0xc00073e800, 0x507, 0x800})
    github.com/docker/buildx/util/imagetools/create.go:221 +0x1a5
 github.com/docker/buildx/util/imagetools.(*Resolver).Combine.func1.1()
    github.com/docker/buildx/util/imagetools/create.go:59 +0x365
 golang.org/x/sync/errgroup.(*Group).Go.func1()
    golang.org/x/sync@v0.2.0/errgroup/errgroup.go:75 +0x64
 created by golang.org/x/sync/errgroup.(*Group).Go
    golang.org/x/sync@v0.2.0/errgroup/errgroup.go:72 +0xa5
crazy-max commented 1 year ago

@jedevc Could this be another containerd issue? :cold_sweat:

crazy-max commented 1 year ago

Maybe related to https://github.com/containerd/containerd/commit/14a08721cc8ea42c7c9059d6266fc807650e2325

jedevc commented 1 year ago

Maybe also related to something like https://github.com/containerd/containerd/pull/8379? This code is kinda intricate, it looks like it's at least around the same kind of place.