"Filesystem layer verification failed for digest" for Ubuntu installation of Docker Engine

marcin-jozwikowski commented 1 year ago

Description

I'm running docker on Ubuntu 22.04 (CLI Engine, not Docker-Desktop) and I keep getting layer verification error. I've tested this command multiple times. This issue occurs on different layers (although some tend to have more occurrences - seems that bigger layers have less chance of success) and once in a while everything works. But on other images (i.e. mcr.microsoft.com/mssql/server:2019-CU18-ubuntu-20.04) I wasn't able to successfully finish the pull.

It's now a fresh installation. I've completely removed Docker along with its configs and reinstalled it all. /var/lib/docker has been removed, so were /etc/docker/daemon.json and ~/.docker/config.json. I don't use proxy, the internet connection is stable, and this is not a VM - it's a native Ubuntu installation. I've tried docker-desktop and there everything is fine, so it's not a repository issue nor a connection one. Those pulls are not from any private repository, nor are those images built by me.

The following two commands were run immediately one after another:

mjozwikowski:~$ docker -D pull postgres:latest
latest: Pulling from library/postgres
648e0aadf75a: Downloading [==========================>                        ]  15.44MB/29.12MB
f715c8c55756: Download complete 
b11a1dc32c8c: Downloading  2.693MB
f29e8ba9d17c: Verifying Checksum 
78af88a8afb0: Waiting 
b74279c188d9: Waiting 
6e3e5bf64fd2: Waiting 
b62a2c2d2ce5: Waiting 
eba91ca3c7a3: Waiting 
d4a24cdf2433: Waiting 
b20f8a8dfd5c: Waiting 
e0731dd084c3: Waiting 
0361da6a228e: Waiting 
filesystem layer verification failed for digest sha256:f29e8ba9d17cfa147141648b72ff8ab49a86234dfe1194f6220690939f1daa3c

mjozwikowski:~$ docker -D pull postgres:latest
latest: Pulling from library/postgres
648e0aadf75a: Pull complete 
f715c8c55756: Pull complete 
b11a1dc32c8c: Pull complete 
f29e8ba9d17c: Pull complete 
78af88a8afb0: Pull complete 
b74279c188d9: Pull complete 
6e3e5bf64fd2: Pull complete 
b62a2c2d2ce5: Pull complete 
eba91ca3c7a3: Verifying Checksum 
d4a24cdf2433: Download complete 
b20f8a8dfd5c: Download complete 
e0731dd084c3: Download complete 
0361da6a228e: Download complete 
filesystem layer verification failed for digest sha256:eba91ca3c7a37844775569d1771c8acfab80b32d9c24f4a0b5b998d91911d747

Reproduce

docker -D pull postgres:latest

Expected behavior

docker pull should complete without any issues

docker version

Client: Docker Engine - Community
 Version:           24.0.5
 API version:       1.43
 Go version:        go1.20.6
 Git commit:        ced0996
 Built:             Fri Jul 21 20:35:18 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.5
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.6
  Git commit:       a61e2b4
  Built:            Fri Jul 21 20:35:18 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.22
  GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client: Docker Engine - Community
 Version:    24.0.5
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.20.2
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 1
 Server Version: 24.0.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.2.0-26-generic
 Operating System: Ubuntu 22.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 42.93GiB
 Name: Company
 ID: 8c0696ec-411e-479d-91b8-82be613dda2f
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

mjozwikowski:~$ sudo service docker restart
mjozwikowski:~$ docker pull postgres
Using default tag: latest
latest: Pulling from library/postgres
648e0aadf75a: Pull complete 
f715c8c55756: Pull complete 
b11a1dc32c8c: Pull complete 
f29e8ba9d17c: Pull complete 
78af88a8afb0: Pull complete 
b74279c188d9: Pull complete 
6e3e5bf64fd2: Pull complete 
b62a2c2d2ce5: Pull complete 
eba91ca3c7a3: Verifying Checksum 
d4a24cdf2433: Download complete 
b20f8a8dfd5c: Download complete 
e0731dd084c3: Download complete 
0361da6a228e: Download complete 
filesystem layer verification failed for digest sha256:eba91ca3c7a37844775569d1771c8acfab80b32d9c24f4a0b5b998d91911d747

Resulted in following entries in journalctl

sie 06 11:33:01 sudo[10242]: mjozwikowski : TTY=pts/0 ; PWD=/home/mjozwikowski ; USER=root ; COMMAND=/usr/sbin/service docker restart
sie 06 11:33:01 dockerd[9480]: time="2023-08-06T11:33:01.889463320+02:00" level=info msg="Processing signal 'terminated'"
sie 06 11:33:01 dockerd[9480]: time="2023-08-06T11:33:01.890225881+02:00" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
sie 06 11:33:01 dockerd[9480]: time="2023-08-06T11:33:01.890540576+02:00" level=info msg="Daemon shutdown complete"
sie 06 11:33:01 systemd[1]: docker.service: Deactivated successfully.
sie 06 11:33:01 systemd[1]: docker.service: Consumed 5.232s CPU time.
sie 06 11:33:01 dockerd[10248]: time="2023-08-06T11:33:01.990678853+02:00" level=info msg="Starting up"
sie 06 11:33:01 dockerd[10248]: time="2023-08-06T11:33:01.991800333+02:00" level=info msg="detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf"
sie 06 11:33:02 dockerd[10248]: time="2023-08-06T11:33:02.042684542+02:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
sie 06 11:33:02 dockerd[10248]: time="2023-08-06T11:33:02.043220655+02:00" level=info msg="Loading containers: start."
sie 06 11:33:02 dockerd[10248]: time="2023-08-06T11:33:02.709110191+02:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
sie 06 11:33:02 dockerd[10248]: time="2023-08-06T11:33:02.800643704+02:00" level=info msg="Loading containers: done."
sie 06 11:33:02 dockerd[10248]: time="2023-08-06T11:33:02.806029624+02:00" level=info msg="Docker daemon" commit=a61e2b4 graphdriver=overlay2 version=24.0.5
sie 06 11:33:02 dockerd[10248]: time="2023-08-06T11:33:02.806068207+02:00" level=info msg="Daemon has completed initialization"
sie 06 11:33:02 dockerd[10248]: time="2023-08-06T11:33:02.819571260+02:00" level=info msg="API listen on /run/docker.sock"
sie 06 11:33:18 dockerd[10248]: time="2023-08-06T11:33:18.791852149+02:00" level=error msg="filesystem layer verification failed for digest sha256:eba91ca3c7a37844775569d1771c8acfab80b32d9c24f4a0b5b998d91911d747"
sie 06 11:33:18 dockerd[10248]: time="2023-08-06T11:33:18.791895641+02:00" level=error msg="Download failed after 1 attempts: filesystem layer verification failed for digest sha256:eba91ca3c7a37844775569d1771c8acfab80b32d9c24f4a0b5b998d91911d747"
sie 06 11:33:18 dockerd[10248]: time="2023-08-06T11:33:18.808820762+02:00" level=info msg="Attempting next endpoint for pull after error: filesystem layer verification failed for digest sha256:eba91ca3c7a37844775569d1771c8acfab80b32d9c24f4a0b5b998d91911d747"
sie 06 11:33:18 dockerd[10248]: time="2023-08-06T11:33:18.810248872+02:00" level=info msg="Layer sha256:e918add37af5039e3aa5c6fc505fbc0d7f3084997afdafe8cfa5cdc60794371f cleaned up"
sie 06 11:33:18 dockerd[10248]: time="2023-08-06T11:33:18.811304838+02:00" level=info msg="Layer sha256:63d0f06fce28434b01c759f220f44e6d18f3820db10e724f1100293d38698df0 cleaned up"
sie 06 11:33:18 dockerd[10248]: time="2023-08-06T11:33:18.870937723+02:00" level=info msg="Layer sha256:22de904444fbf7bb0f4598f30014ab426ed2e4038bb8eae707c5a503c33a4a72 cleaned up"
sie 06 11:33:18 dockerd[10248]: time="2023-08-06T11:33:18.870958051+02:00" level=info msg="Layer sha256:a067415fc97316aa484b04343164efd15f2de48522fa5ccf8ae33cfe7b91b2d4 cleaned up"
sie 06 11:33:18 dockerd[10248]: time="2023-08-06T11:33:18.870962900+02:00" level=info msg="Layer sha256:a3e038b9513aad18e436f7ed38244eabdb2eb15655212a48428dcf96af4aecf6 cleaned up"
sie 06 11:33:18 dockerd[10248]: time="2023-08-06T11:33:18.870966437+02:00" level=info msg="Layer sha256:1ae5138983888dd7346b42ca998c986e8f75113599395b89f2294652f9fb7e76 cleaned up"
sie 06 11:33:18 dockerd[10248]: time="2023-08-06T11:33:18.870970124+02:00" level=info msg="Layer sha256:d33250e6749e53e7280f627466421a672cfa59f10a66cd654830240d1387b816 cleaned up"
sie 06 11:33:18 dockerd[10248]: time="2023-08-06T11:33:18.870973831+02:00" level=info msg="Layer sha256:c6e34807c2d51444c41c15f4fda65847faa2f43c9b4b976a2f6f476eca7429ce cleaned up"

thaJeztah commented 1 year ago

This error would occur when validating the checksum of the extracted files doesn't match the expected checksum 🤔. No direct clues, other than actual disk issues.

Is there anything special with /var/lib/docker, e.g., could it be a separate mount for storage or something along those lines?
Is there any-virus / malware software running on the machine? I have seen some reports where such software was quarantining files, causing validation to fail.

marcin-jozwikowski commented 1 year ago

No. /var/lib is part of / which takes the whole /dev/nvme0n1p5 formatted to Ext4 (version 1.0).

mjozwikowski:~$ df -ha
df: /run/user/1000/doc: Operation not permitted
Filesystem                   Size  Used Avail Use% Mounted on
sysfs                           0     0     0    - /sys
proc                            0     0     0    - /proc
udev                          22G     0   22G   0% /dev
devpts                          0     0     0    - /dev/pts
tmpfs                        4,3G  3,0M  4,3G   1% /run
/dev/nvme0n1p5               359G  144G  197G  43% /
securityfs                      0     0     0    - /sys/kernel/security
tmpfs                         22G   51M   22G   1% /dev/shm
tmpfs                        5,0M  4,0K  5,0M   1% /run/lock
cgroup2                         0     0     0    - /sys/fs/cgroup
pstore                          0     0     0    - /sys/fs/pstore
efivarfs                        0     0     0    - /sys/firmware/efi/efivars
bpf                             0     0     0    - /sys/fs/bpf
systemd-1                       -     -     -    - /proc/sys/fs/binfmt_misc
mqueue                          0     0     0    - /dev/mqueue
hugetlbfs                       0     0     0    - /dev/hugepages
debugfs                         0     0     0    - /sys/kernel/debug
tracefs                         0     0     0    - /sys/kernel/tracing
fusectl                         0     0     0    - /sys/fs/fuse/connections
configfs                        0     0     0    - /sys/kernel/config
ramfs                           0     0     0    - /run/credentials/systemd-sysusers.service
/dev/loop0                   128K  128K     0 100% /snap/bare/5
/dev/loop1                   119M  119M     0 100% /snap/core/15419
/dev/loop2                   119M  119M     0 100% /snap/core/15511
/dev/loop3                    56M   56M     0 100% /snap/core18/2751
/dev/loop4                    56M   56M     0 100% /snap/core18/2785
/dev/loop5                    64M   64M     0 100% /snap/core20/1950
/dev/loop7                    74M   74M     0 100% /snap/core22/817
/dev/loop6                    64M   64M     0 100% /snap/core20/1974
/dev/loop8                    74M   74M     0 100% /snap/core22/858
/dev/loop10                  238M  238M     0 100% /snap/firefox/2971
/dev/loop11                  219M  219M     0 100% /snap/gnome-3-34-1804/90
/dev/loop12                  219M  219M     0 100% /snap/gnome-3-34-1804/93
/dev/loop13                  350M  350M     0 100% /snap/gnome-3-38-2004/140
/dev/loop14                  350M  350M     0 100% /snap/gnome-3-38-2004/143
/dev/loop15                  467M  467M     0 100% /snap/gnome-42-2204/111
/dev/loop16                  486M  486M     0 100% /snap/gnome-42-2204/120
/dev/loop17                   82M   82M     0 100% /snap/gtk-common-themes/1534
/dev/loop18                   92M   92M     0 100% /snap/gtk-common-themes/1535
/dev/loop19                  9,8M  9,8M     0 100% /snap/htop/3735
/dev/loop20                  9,8M  9,8M     0 100% /snap/htop/3758
/dev/loop21                   38M   38M     0 100% /snap/hunspell-dictionaries-1-7-2004/2
/dev/loop22                  437M  437M     0 100% /snap/kde-frameworks-5-96-qt-5-15-5-core20/7
/dev/loop23                  261M  261M     0 100% /snap/kde-frameworks-5-core18/32
/dev/loop24                  290M  290M     0 100% /snap/kde-frameworks-5-core18/35
/dev/loop25                  449M  449M     0 100% /snap/kf5-5-104-qt-5-15-8-core22/7
/dev/loop26                  449M  449M     0 100% /snap/kf5-5-104-qt-5-15-8-core22/9
/dev/loop27                  253M  253M     0 100% /snap/krita/85
/dev/loop28                  253M  253M     0 100% /snap/krita/90
/dev/loop29                  113M  113M     0 100% /snap/slack/82
/dev/loop30                  114M  114M     0 100% /snap/slack/83
/dev/loop31                   46M   46M     0 100% /snap/snap-store/638
/dev/loop32                   13M   13M     0 100% /snap/snap-store/959
/dev/loop33                   54M   54M     0 100% /snap/snapd/19361
/dev/loop34                   54M   54M     0 100% /snap/snapd/19457
/dev/loop35                  512K  512K     0 100% /snap/snapd-desktop-integration/57
/dev/loop36                  512K  512K     0 100% /snap/snapd-desktop-integration/83
/dev/loop37                   66M   66M     0 100% /snap/sublime-text/118
/dev/loop38                   64M   64M     0 100% /snap/sublime-text/122
/dev/nvme0n1p5               359G  144G  197G  43% /var/snap/firefox/common/host-hunspell
/dev/loop39                  254M  254M     0 100% /snap/subsync/11
/dev/nvme0n1p1               256M   72M  185M  29% /boot/efi
tmpfs                         22G     0   22G   0% /run/qemu
binfmt_misc                     0     0     0    - /proc/sys/fs/binfmt_misc
sunrpc                          0     0     0    - /run/rpc_pipefs
tmpfs                        4,3G  3,0M  4,3G   1% /run/snapd/ns
nsfs                            0     0     0    - /run/snapd/ns/snapd-desktop-integration.mnt
tmpfs                        4,3G  136K  4,3G   1% /run/user/1000
/home/mjozwikowski/.Private  359G  144G  197G  43% /home/mjozwikowski
nsfs                            0     0     0    - /run/snapd/ns/snap-store.mnt
/dev/loop41                  238M  238M     0 100% /snap/firefox/2987
nsfs                            0     0     0    - /run/snapd/ns/firefox.mnt

And the whole drive seems to be in good condition:

mjozwikowski:~$ sudo nvme smart-log /dev/nvme0
Smart Log for NVME device:nvme0 namespace-id:ffffffff
critical_warning            : 0
temperature             : 24 C (297 Kelvin)
available_spare             : 100%
available_spare_threshold       : 10%
percentage_used             : 3%
endurance group critical warning summary: 0
data_units_read             : 18 308 944
data_units_written          : 25 745 218
host_read_commands          : 264 878 067
host_write_commands         : 429 459 124
controller_busy_time            : 4 131
power_cycles                : 1 547
power_on_hours              : 3 531
unsafe_shutdowns            : 27
media_errors                : 0
num_err_log_entries         : 0
Warning Temperature Time        : 0
Critical Composite Temperature Time : 0
Thermal Management T1 Trans Count   : 0
Thermal Management T2 Trans Count   : 0
Thermal Management T1 Total Time    : 0
Thermal Management T2 Total Time    : 0

No, I don't have any antivirus/antimalware software installed. Unless something is one and I'm not aware... At least I didn't until today. Installed clamav, and did a full scan - nothing found.

Would there be a way to enable some more detailed logs? I'd gladly send you all the details I could get but I don't know what exactly to do.

FzzfSisyphus commented 7 months ago

I meet same issue, if you have enough disk space check your internet connection, I am using a ASUS TUF labtop after I switch from WIFI to wired connect I solve the problem.

rruales6 commented 4 months ago

can relate that changing from wired to wireless solved my issue

marcin-jozwikowski commented 4 months ago

In my case it was the other way around. Wired connection seemed more reliable back then. I ended up doing a warranty repair for the laptop I was working on. They replaced the whole motherboard (which has almost everything on it) - only RAM and hard drive remained the same. After that everything worked perfectly - even without reinstalling the OS. It's hard for me to even guess what could have been the issue.

moby / moby