search

pop-os / pop

A project for managing all Pop!_OS sources
https://system76.com/pop
2.48k stars 87 forks source link

overlayfs missing after kernel upgrade to 6.9.3-76060903.202405300957~1726766035~22.04~4092a0e #3373

Closed nevillelyh closed 2 months ago

nevillelyh commented 2 months ago

Distribution (run cat /etc/os-release):

 NAME="Pop!_OS"
VERSION="22.04 LTS"
ID=pop
ID_LIKE="ubuntu debian"
PRETTY_NAME="Pop!_OS 22.04 LTS"
VERSION_ID="22.04"
HOME_URL="https://pop.system76.com"
SUPPORT_URL="https://support.system76.com"
BUG_REPORT_URL="https://github.com/pop-os/pop/issues"
PRIVACY_POLICY_URL="https://system76.com/privacy"
VERSION_CODENAME=jammy
UBUNTU_CODENAME=jammy
LOGO=distributor-logo-pop-os

Related Application and/or Package Version (run apt policy $PACKAGE NAME):

❯ apt policy linux-image-6.9.3-76060903-generic
linux-image-6.9.3-76060903-generic:
  Installed: 6.9.3-76060903.202405300957~1726766035~22.04~4092a0e
  Candidate: 6.9.3-76060903.202405300957~1726766035~22.04~4092a0e
  Version table:
 *** 6.9.3-76060903.202405300957~1726766035~22.04~4092a0e 1001
       1001 http://apt.pop-os.org/release jammy/main amd64 Packages
        100 /var/lib/dpkg/status

Issue/Bug Description:

After a kernel update this morning, Docker no longer works and is complaining about missing overlayfs.

journalctl -u docker
Sep 20 17:15:16 borealis systemd[1]: Starting Docker Application Container Engine...
Sep 20 17:15:16 borealis dockerd[2159]: time="2024-09-20T17:15:16.776697770-04:00" level=info msg="Starting up"
Sep 20 17:15:16 borealis dockerd[2159]: time="2024-09-20T17:15:16.777055598-04:00" level=info msg="detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/sys>
Sep 20 17:15:16 borealis dockerd[2159]: time="2024-09-20T17:15:16.845202568-04:00" level=error msg="failed to mount overlay: no such device" storage-driver=overlay2
Sep 20 17:15:16 borealis dockerd[2159]: time="2024-09-20T17:15:16.845215573-04:00" level=error msg="[graphdriver] prior storage driver overlay2 failed: driver not supported"
Sep 20 17:15:16 borealis dockerd[2159]: failed to start daemon: error initializing graphdriver: driver not supported
Sep 20 17:15:16 borealis systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Sep 20 17:15:16 borealis systemd[1]: docker.service: Failed with result 'exit-code'.
Sep 20 17:15:16 borealis systemd[1]: Failed to start Docker Application Container Engine.
Sep 20 17:15:18 borealis systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
Sep 20 17:15:18 borealis systemd[1]: Stopped Docker Application Container Engine.
Sep 20 17:15:18 borealis systemd[1]: docker.service: Start request repeated too quickly.
Sep 20 17:15:18 borealis systemd[1]: docker.service: Failed with result 'exit-code'.
Sep 20 17:15:18 borealis systemd[1]: Failed to start Docker Application Container Engine.

I checked that there's no overlay module, file system and modprobe fails.

❯ lsmod | grep overlay
❯ cat /proc/filesystems | grep overlay
❯ sudo modprobe overlay
modprobe: ERROR: could not insert 'overlay': Invalid argument

dmesg:

[ 1487.040875] BPF:      type_id=20 bits_offset=256
[ 1487.040880] BPF:
[ 1487.040882] BPF: Invalid name
[ 1487.040883] BPF:
[ 1487.040886] failed to validate module [overlay] BTF: -22
[ 1490.886816] kauditd_printk_skb: 4625 callbacks suppressed

But the module file exists.

❯ sudo modinfo /lib/modules/6.9.3-76060903-generic/kernel/fs/overlayfs/overlay.ko.zst
filename:       /lib/modules/6.9.3-76060903-generic/kernel/fs/overlayfs/overlay.ko.zst
alias:          fs-overlay
license:        GPL
description:    Overlay filesystem
author:         Miklos Szeredi <miklos@szeredi.hu>
srcversion:     6F6A12D03FE472B4DC4CE05
depends:
retpoline:      Y
intree:         Y
name:           overlay
vermagic:       6.9.3-76060903-generic SMP preempt mod_unload modversions
sig_id:         PKCS#7
signer:         Build time autogenerated kernel key
sig_key:        78:A9:D4:70:D0:0C:7A:A5:3D:4B:FB:27:ED:42:A2:9A:49:47:C5:26
sig_hashalgo:   sha512
signature:      5B:E9:01:B5:B7:82:69:DE:49:87:75:80:DB:1F:D1:C4:C6:7D:A9:ED:
                EA:38:8C:29:48:A6:0C:AD:6D:8C:2E:CE:AF:54:34:07:00:5D:76:F0:
                36:39:34:30:6C:A6:B3:C3:7D:88:2E:59:9C:B1:43:1C:7B:F8:84:B6:
                1A:00:E7:61:24:E1:7E:AD:E1:CC:E8:09:9F:1A:B4:B3:2F:94:CF:67:
                D8:57:20:3F:03:05:34:B8:ED:9C:3C:9C:7D:2A:5D:86:C0:EA:A4:96:
                77:12:7D:EB:36:3E:1C:B9:68:A3:1C:9D:12:36:CB:20:00:3A:EE:B7:
                B2:64:D9:BD:E3:82:79:F8:9A:09:B4:A8:AF:39:47:33:BB:59:B0:E4:
                B8:F2:D5:3F:A9:EF:D5:B8:14:1E:09:3D:89:AE:B2:1C:46:08:A4:5B:
                D8:46:DA:C6:8C:54:D6:BA:A3:04:FA:DA:E2:A7:97:F1:73:FF:86:4F:
                5B:5A:CE:35:86:BD:6B:1E:37:89:2E:B0:F5:4A:6A:4C:B0:2B:27:14:
                54:D6:DF:70:D7:3F:C7:20:B4:A3:DD:BC:28:C2:77:73:2F:0A:A7:A6:
                94:CC:B3:42:71:F0:36:C5:3D:72:00:45:E0:55:3D:51:C0:CF:00:0B:
                3B:0A:99:07:4D:10:9F:2E:57:18:13:3A:E8:86:D8:8E:08:C3:63:3F:
                EC:05:1C:8A:49:69:9F:F0:A3:F9:14:14:31:81:CB:A0:7A:7B:4F:14:
                18:D5:24:33:57:B2:1D:33:2F:53:B9:68:83:23:25:0D:37:67:A2:B7:
                4D:36:6E:EA:4F:E8:22:E8:4D:30:74:19:CB:42:3D:8D:08:C5:F9:06:
                2E:22:0D:F6:AB:F8:68:69:98:41:4A:1E:76:41:1D:E4:67:46:D3:62:
                28:BE:97:AE:54:D6:04:1B:52:F2:9E:A1:56:AB:EC:14:22:CB:83:58:
                6D:8A:91:60:B4:25:2E:C9:86:9F:7F:63:F3:8E:C6:09:A0:17:E2:E4:
                1C:FD:BC:D6:7E:A4:3A:48:41:EF:B3:01:95:AA:35:48:94:9E:B2:25:
                62:6E:C2:14:00:D5:39:5B:94:0A:BB:0A:DB:32:E7:66:6B:71:00:EA:
                A2:28:EC:B6:2B:3D:A9:0A:91:34:F3:5D:06:32:08:F2:AD:14:17:59:
                93:0C:AB:16:E3:7C:FE:BB:59:2A:6D:9F:59:E6:81:48:10:3B:B6:0C:
                7D:1E:FB:69:AF:76:9B:6F:F3:F6:D7:75:C0:05:87:97:77:73:AB:9D:
                67:FC:47:F7:06:62:DB:CB:FF:A5:08:60:03:F6:AE:7A:4B:20:61:49:
                23:73:42:92:7F:92:4E:DF:FA:AF:4F:48
parm:           redirect_dir:Default to on or off for the redirect_dir feature (bool)
parm:           redirect_always_follow:Follow redirects even if redirect_dir feature is turned off (bool)
parm:           xino_auto:Auto enable xino feature (bool)
parm:           index:Default to on or off for the inodes index feature (bool)
parm:           nfs_export:Default to on or off for the NFS export feature (bool)
parm:           metacopy:Default to on or off for the metadata only copy up feature (bool)
parm:           check_copy_up:Obsolete; does nothing
parm:           redirect_max:Maximum length of absolute redirect xattr value (ushort)

Also this section from /var/log/apt/history.log:

Start-Date: 2024-09-20  10:40:05
Commandline: packagekit role='update-packages'
Requested-By: neville (1000)
Upgrade: libreoffice-l10n-en-gb:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), libreoffice-l10n-en-za:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), linux-system76
:amd64 (6.9.3.76060903.202405300957~1721174657~22.04~abb7c06, 6.9.3.76060903.202405300957~1726766035~22.04~4092a0e), libreoffice-calc:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04
.7), docker-compose-plugin:amd64 (2.29.2-1~ubuntu.22.04~jammy, 2.29.6-1~ubuntu.22.04~jammy), libreoffice-gnome:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), libmm-glib0:amd64
(1.20.0-1~ubuntu22.04.3, 1.20.0-1~ubuntu22.04.4), uno-libs-private:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), libreoffice-base-core:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7
-0ubuntu0.22.04.7), docker-ce-cli:amd64 (5:27.2.1-1~ubuntu.22.04~jammy, 5:27.3.0-1~ubuntu.22.04~jammy), linux-headers-generic:amd64 (6.9.3.76060903.202405300957~1721174657~22.04~abb7c06, 6.
9.3.76060903.202405300957~1726766035~22.04~4092a0e), libreoffice-core:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), modemmanager:amd64 (1.20.0-1~ubuntu22.04.3, 1.20.0-1~ubuntu
22.04.4), libreoffice-common:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), ure:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), linux-headers-6.9.3-76060903-generic
:amd64 (6.9.3-76060903.202405300957~1721174657~22.04~abb7c06, 6.9.3-76060903.202405300957~1726766035~22.04~4092a0e), linux-headers-6.9.3-76060903:amd64 (6.9.3-76060903.202405300957~17211746
57~22.04~abb7c06, 6.9.3-76060903.202405300957~1726766035~22.04~4092a0e), libreoffice-draw:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), libuno-purpenvhelpergcc3-3:amd64 (1:7.3
.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), libreoffice-help-pt-br:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), libuno-cppu3:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubu
ntu0.22.04.7), libreoffice-impress:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), libreoffice-l10n-ar:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), libreoffice-l1
0n-de:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), libreoffice-l10n-es:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), libreoffice-l10n-fr:amd64 (1:7.3.7-0ubuntu0
.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), libreoffice-l10n-it:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), libreoffice-l10n-ja:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22
.04.7), libreoffice-l10n-pt:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), libreoffice-l10n-ru:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), libuno-cppuhelpergcc3
-3:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), fonts-opensymbol:amd64 (2:102.12+LibO7.3.7-0ubuntu0.22.04.6, 2:102.12+LibO7.3.7-0ubuntu0.22.04.7), linux-generic:amd64 (6.9.3.
76060903.202405300957~1721174657~22.04~abb7c06, 6.9.3.76060903.202405300957~1726766035~22.04~4092a0e), docker-buildx-plugin:amd64 (0.16.2-1~ubuntu.22.04~jammy, 0.17.1-1~ubuntu.22.04~jammy),
 libreoffice-l10n-pt-br:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), docker-ce:amd64 (5:27.2.1-1~ubuntu.22.04~jammy, 5:27.3.0-1~ubuntu.22.04~jammy), libreoffice-help-zh-cn:am
d64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), libreoffice-help-zh-tw:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), libpcap0.8:amd64 (1.10.1-4build1, 1.10.1-4ubuntu
1.22.04.1), libreoffice-style-colibre:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), libreoffice-writer:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), libuno-salhe
lpergcc3-3:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), libreoffice-l10n-zh-cn:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), libreoffice-l10n-zh-tw:amd64 (1:7.3
.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), docker-ce-rootless-extras:amd64 (5:27.2.1-1~ubuntu.22.04~jammy, 5:27.3.0-1~ubuntu.22.04~jammy), libreoffice-help-de:amd64 (1:7.3.7-0ubuntu0.2
2.04.6, 1:7.3.7-0ubuntu0.22.04.7), libreoffice-help-es:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), libreoffice-help-fr:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.0
4.7), libreoffice-help-it:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), libreoffice-help-ja:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), libreoffice-help-pt:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), libreoffice-help-ru:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), linux-image-generic:amd64 (6.9.3.76060903.202405300957~1721174657~22.04~abb7c06, 6.9.3.76060903.202405300957~1726766035~22.04~4092a0e), libreoffice-help-common:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), python3-uno:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), libuno-sal3:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), linux-modules-6.9.3-76060903-generic:amd64 (6.9.3-76060903.202405300957~1721174657~22.04~abb7c06, 6.9.3-76060903.202405300957~1726766035~22.04~4092a0e), libreoffice-style-yaru:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), libreoffice-math:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), libreoffice-gtk3:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), linux-image-6.9.3-76060903-generic:amd64 (6.9.3-76060903.202405300957~1721174657~22.04~abb7c06, 6.9.3-76060903.202405300957~1726766035~22.04~4092a0e), libreoffice-help-en-gb:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), libreoffice-help-en-us:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), libreoffice-style-elementary:amd64 (1:7.3.7-0ubuntu0.22.04.6, 1:7.3.7-0ubuntu0.22.04.7), linux-libc-dev:amd64 (6.9.3-76060903.202405300957~1721174657~22.04~abb7c06, 6.9.3-76060903.202405300957~1726766035~22.04~4092a0e)
End-Date: 2024-09-20  10:41:02

Steps to reproduce (if you know):

Expected behavior:

Other Notes:

nevillelyh commented 2 months ago

Turns out my /boot/efi is on /dev/nvme[01]n1p1 manually mirrored but not RAID1, and they're out of sync after update. rsyncing the partitions fixed the issue. Closing this as a false alarm.