[BUG] failed to solve: failed to walk: resolve : lstat /var/lib/docker/overlay2/diff: no such file or directory

[WolfgangFahl]

Description

https://github.com/docker/for-mac/issues/1396 has been discussed since 2017 and was closed for no proper reason. The problem has shown up dozen of times on macs and linux machines. It happened to me today.

Steps To Reproduce

On my Ubunut 22.04 LTS machine Linux arche 5.15.0-60-generic #66-Ubuntu SMP Fri Jan 20 14:29:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux Docker version 23.0.0, build e92dd87

I did a pip install pymediawikidocker mwcluster -f

which runs happily in my CI of the project https://github.com/WolfgangFahl/pymediawikidocker but fails locally.

The necessary docker files are automatically generated.

Trying a docker build to reproduce:

docker compose build
[+] Building 2.5s (6/22)                                                                                  
 => [internal] load build definition from Dockerfile                                                 0.1s
 => => transferring dockerfile: 2.25kB                                                               0.0s
 => [internal] load .dockerignore                                                                    0.1s
 => => transferring context: 2B                                                                      0.0s
 => [internal] load metadata for docker.io/library/mediawiki:1.27.7                                  1.8s
 => CACHED [ 1/18] FROM docker.io/library/mediawiki:1.27.7@sha256:2c46c500b7191763b30b0a51cbed87d14  0.0s
 => ERROR [internal] load build context                                                              0.0s
 => => transferring context: 20B                                                                     0.0s
 => CANCELED [ 2/18] RUN apt-get -y update &&     apt-get -y --no-install-recommends --fix-missing   0.5s
------
 > [internal] load build context:
------
failed to solve: failed to walk: resolve : lstat /var/lib/docker/overlay2/diff: no such file or directory

pointing to https://github.com/docker/for-mac/issues/1396 which is docker for mac specific and very very unfortunately closed only because there has been no new comments recently (which seems to have an assumption that bugs are fixed by waiting for nobody to complain any more ...)

Compose Version

Docker Compose version v2.15.1
docker-compose version 1.29.2, build unknown
docker-py version: 5.0.3
CPython version: 3.10.6
OpenSSL version: OpenSSL 3.0.2 15 Mar 2022

Docker Environment

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.15.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-scan

Server:
 Containers: 2
  Running: 2
  Paused: 0
  Stopped: 0
 Images: 24
 Server Version: 23.0.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 31aa4358a36870b21a992d3ad2bef29e1d693bec
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-60-generic
 Operating System: Ubuntu 22.04.1 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 31.25GiB
 Name: arche
 ID: TXHV:IAOH:IDIB:RDBS:SIAT:7F6A:B3ZE:XO35:VUY7:4DOZ:4LUW:ZNHQ
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

Please switch off the bot that closed issues that have no comments e.g. if the issues has previously had a certain amount of activity indicating that the problem is indeed common.

[dokuboyejo]

Just noticed the issue now as well... what helped was umount /var/lib/docker/overlay2 and then restart docker service

[robertvanlienden]

Fixed by running docker system prune -a as mentioned here

[ndeloof]

Please try running a plain docker build with Dockerfile(s) from your compose file (docker build ...) to confirm this is a Compose issue, otherwise issue should be reported to docker/buildx repository if this is a general build issue.

[WolfgangFahl]

Unfortunately the reproducatbility of this bug is limited and therefore the causes are still blury and need more investigation and feedback. This shouldn't be a reason to close the issue though.

[ndeloof]

sure, but we first need to confirm this is a compose issue, and not a build/engine/docker desktop one, otherwise you won't get the right team investigating

[RuslanUshakov]

the same stuff for me, face with the issue as well

[kalenjohnson]

Fixed by running docker system prune -a as mentioned here

Fixed the issue for me :+1:

[bluisier]

In my case, I did not need to do a docker system prune -a, removing the build cache with docker builder prune -a was enough, as mentioned here.

[neersighted]

Based on the symptoms and the mitigation (bulider prune -a), this looks like https://github.com/moby/moby/issues/44943 and can be closed if it does not reproduce on 23.0.1.

[neersighted]

Closing for now as this has not been reproduced on subsequent versions.

[ssichynskyi]

If nothing from above helps, on linux distro (no docker desktop), do the following to do kinda factory reset for docker

• # systemctl stop docker
• # rm -rf /var/lib/docker
• # systemctl start docker