docker push fails with error creating overlay mount to /var/lib/docker/overlay2/.../merged: device or resource busy

[clauderobi]

Description

When running docker push on a raspberry I get a lot of failure. Eventually, after many attempts, a push completes successfully.

Steps to reproduce the issue:

1. Create an image, consisting of many layers, large and small
2. Use docker push to a LAN registry

Describe the results you received:

error creating overlay mount to /var/lib/docker/overlay2/..../merged: device or resource busy

For each attempt, some layer(s) do get pushed successfully so eventually only 1 layer remains at which point in time the attempt will succeed.

Describe the results you expected:

A successful push to the registry

Additional information you deem important (e.g. issue happens only occasionally):

The same type of operation to the same registry on a X86 always works the first time. The version on the X86 is 18.09.7

Output of docker version:

Client: Docker Engine - Community Version: 19.03.1 API version: 1.40 Go version: go1.12.5 Git commit: 74b1e89 Built: Thu Jul 25 21:33:17 2019 OS/Arch: linux/arm Experimental: true

Server: Docker Engine - Community Engine: Version: 19.03.1 API version: 1.40 (minimum version 1.12) Go version: go1.12.5 Git commit: 74b1e89 Built: Thu Jul 25 21:27:09 2019 OS/Arch: linux/arm Experimental: false containerd: Version: 1.2.6 GitCommit: 894b81a4b802e4eb2a91d1ce216b8817763c29fb runc: Version: 1.0.0-rc7+dev GitCommit: 029124da7af7360afa781a0234d1b083550f797c docker-init: Version: 0.18.0 GitCommit: fec3683

Output of docker info:

Client: Debug Mode: false Plugins: app: Docker Application (Docker Inc., v0.8.0) buildx: Build with BuildKit (Docker Inc., v0.2.2-10-g3f18b65-tp-docker)

Server: Containers: 3 Running: 2 Paused: 0 Stopped: 1 Images: 76 Server Version: 19.03.1 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: active NodeID: mfyd6je7xks6pi51rz0oamcbk Is Manager: false Node Address: 192.168.31.31 Manager Addresses: 192.168.31.20:2377 192.168.31.8:2377 Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb runc version: 029124da7af7360afa781a0234d1b083550f797c init version: fec3683 Security Options: seccomp Profile: default Kernel Version: 4.19.58+ Operating System: Raspbian GNU/Linux 10 (buster) OSType: linux Architecture: armv6l CPUs: 1 Total Memory: 432.7MiB Name: raspberrypi ID: 4LJG:SJWE:7RKA:GWJV:4N32:FUDU:TW4P:CEJY:ZRTL:5VPI:TUWS:YAYU Docker Root Dir: /var/lib/docker Debug Mode: false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: gitmaster.cbcr.me:5000 127.0.0.0/8 Live Restore Enabled: false

WARNING: No swap limit support WARNING: No cpu cfs quota support WARNING: No cpu cfs period support WARNING: No cpuset support

Additional environment details (AWS, VirtualBox, physical, etc.):

A sample output:

The push refers to repository [gitmaster.cbcr.me/nodemngr-armv6l] d7df498795d7: Pushing [==================================================>] 13.25kB/13.25kB 920c84a5862a: Pushed 32170657d427: Layer already exists 55f7e66e9106: Pushed ce98bd33fd42: Pushing [=================> ] 8.713MB/25.01MB bee8fc9dc044: Layer already exists b04951da02ad: Layer already exists b35fac3d7190: Layer already exists 91af7c91711f: Layer already exists 22de177773d5: Layer already exists 9be8c5a7378e: Layer already exists 83a839f8b4fb: Layer already exists 3b565f87592e: Layer already exists 31b048eae07b: Layer already exists 5fb20b68e070: Layer already exists 5743fac7d76e: Layer already exists bd04091dc8fc: Layer already exists error creating overlay mount to /var/lib/docker/overlay2/fc59a82445daa04d14730b29ff96664ed69d92e4a8c5bc1d57ef3a67c350fe61/merged: device or resource busy

[maaikez]

https://github.com/docker/for-linux/issues/711

[thaJeztah]

This should be fixed by https://github.com/moby/moby/pull/39780, which is included in docker 18.09.10 and up (through https://github.com/docker/engine/pull/333), and docker 19.03.3 and up (through https://github.com/docker/engine/pull/332)

[daliborfilus]

@thaJeztah I just encountered this on Docker CE 19.03.5, linux 4.19.0-6-amd64:

ERROR: Job failed (system failure): Error response from daemon: error creating overlay mount to /var/lib/docker/overlay2/2ab4726161cb6f095f04dc13a788c9db6507f532cc31275888954c7994473230/merged: device or resource busy (executor_docker.go:742:0s)

EDIT: as I see in the issue docker/for-linux#711, I'm not the first. But the comments suggests doing docker system prune first.

[thaJeztah]

What is executor_docker.go ? Is that a tool you're using?

[daliborfilus]

The message is copied from Gitlab CI Runner, which is written in Go. Please ignore that part.