search

clearlinux / distribution

Placeholder repository to allow filing of general bugs/issues/etc against the Clear Linux OS for Intel Architecture linux distribution
521 stars 29 forks source link

clr-boot-manager list-kernels segmentation fault on 36380 kvm kernel #2636

Closed cphuntington97 closed 2 years ago

cphuntington97 commented 2 years ago 
nnn@www~ $ swupd info
Distribution:      Clear Linux OS
Installed version: 36380
Version URL:       https://cdn.download.clearlinux.org/update
Content URL:       https://cdn.download.clearlinux.org/update
nnn@www~ $ uname -r
5.13.19-547.kvm
nnn@www~ $ sudo clr-boot-manager list-kernels
Password: 
Segmentation fault

I'm not sure if this bug has any relationship to the kvm kernel being several versions behind the native kernel.

cphuntington97 commented 2 years ago

More info: segmentation fault happens on the native kernel too. I think it has something to do with the version.

nanon@hkg~ $ swupd info
Distribution:      Clear Linux OS
Installed version: 36400
Version URL:       https://cdn.download.clearlinux.org/update
Content URL:       https://cdn.download.clearlinux.org/update
nanon@hkg~ $ uname -r
5.13.13-1070.native
nanon@hkg~ $ sudo clr-boot-manager list-kernels
Segmentation fault
nanon@hkg~ $

On my desktop with the desktop bundle installed, I have kernel version 5.16.17-1136.native On my server without the desktop bundle, I have kernel version 5.13.13-1070.native

thiagomacieira commented 2 years ago

Cannot confirm.

# swupd info
Distribution:      Clear Linux OS
Installed version: 36400
Version URL:       https://cdn.download.clearlinux.org/update
Content URL:       https://cdn.download.clearlinux.org/update
# clr-boot-manager list-kernels
* org.clearlinux.native.5.17.9-1145
  org.clearlinux.native.5.17.3-1139
cphuntington97 commented 2 years ago

Following is information about 3 machines with coredumps.

win@cnx~ $ clr-network-troubleshooter
clr-network-troubleshooter v15
Running on Clear Linux release 36400
---------------------------------

Testing https site: [curl --connect-timeout 10 -s -i -L https://download.clearlinux.org/latest]
  ✓ Got release 36400

Testing http site: [curl --connect-timeout 10 -s -i -L http://neverssl.com/]
  ✓ OK

=====================
Connectivity seems ok
win@cnx~ $ sudo clr-boot-manager list-kernels
Password: 
Segmentation fault
win@cnx~ $ uname -r
5.16.17-1136.native
win@cnx~ $ ls -al /usr/lib/kernel
total 127768
drwxr-xr-x  3 root root     4096 May 23 15:25 .
drwxr-xr-x 50 root root     4096 May 24 21:11 ..
-rw-r--r--  1 root root      267 Mar 17 15:42 cmdline-5.16.15-1134.native
-rw-r--r--  1 root root      267 Mar 24 12:21 cmdline-5.16.17-1136.native
-rw-r--r--  1 root root      267 Mar 29 16:32 cmdline-5.16.18-1137.native
-rw-r--r--  1 root root      267 Apr  6 12:00 cmdline-5.16.18-1138.native
-rw-r--r--  1 root root      267 Apr 14 20:00 cmdline-5.17.3-1139.native
-rw-r--r--  1 root root      267 Apr 22 06:17 cmdline-5.17.4-1139.native
-rw-r--r--  1 root root      267 May 10 14:58 cmdline-5.17.6-1141.native
-rw-r--r--  1 root root      267 May 12 19:26 cmdline-5.17.7-1142.native
-rw-r--r--  1 root root      267 May 17 21:04 cmdline-5.17.8-1143.native
-rw-r--r--  1 root root      267 May 18 06:09 cmdline-5.17.9-1144.native
-rw-r--r--  1 root root      267 May 23 15:22 cmdline-5.17.9-1145.native
-rw-r--r--  1 root root   232392 Mar 17 15:42 config-5.16.15-1134.native
-rw-r--r--  1 root root   232392 Mar 23 06:06 config-5.16.17-1136.native
-rw-r--r--  1 root root   232392 Mar 28 06:06 config-5.16.18-1137.native
-rw-r--r--  1 root root   232416 Apr  6 12:04 config-5.16.18-1138.native
-rw-r--r--  1 root root   234531 Apr 14 20:00 config-5.17.3-1139.native
-rw-r--r--  1 root root   234489 Apr 25 14:15 config-5.17.4-1139.native
-rw-r--r--  1 root root   234474 May 10 14:58 config-5.17.6-1141.native
-rw-r--r--  1 root root   234474 May 12 19:26 config-5.17.7-1142.native
-rw-r--r--  1 root root   234474 May 17 21:04 config-5.17.8-1143.native
-rw-r--r--  1 root root   234474 May 18 06:09 config-5.17.9-1144.native
-rw-r--r--  1 root root   234463 May 23 15:22 config-5.17.9-1145.native
lrwxrwxrwx  1 root root       33 May 20 10:48 default-native -> org.clearlinux.native.5.17.9-1145
drwxr-xr-x  2 root root     4096 Apr  4  2021 install.d
-rwxr-xr-x  1 root root 11611872 Mar 16 10:07 org.clearlinux.native.5.16.15-1134
-rwxr-xr-x  1 root root 11611232 Mar 23 06:06 org.clearlinux.native.5.16.17-1136
-rwxr-xr-x  1 root root 11609408 Mar 28 06:06 org.clearlinux.native.5.16.18-1137
-rwxr-xr-x  1 root root 11590976 Apr  6 12:04 org.clearlinux.native.5.16.18-1138
-rwxr-xr-x  1 root root 11675552 Apr 14 20:00 org.clearlinux.native.5.17.3-1139
-rwxr-xr-x  1 root root 11672672 Apr 25 14:15 org.clearlinux.native.5.17.4-1139
-rwxr-xr-x  1 root root 11674816 May  9 14:11 org.clearlinux.native.5.17.6-1141
-rwxr-xr-x  1 root root 11674464 May 12 07:07 org.clearlinux.native.5.17.7-1142
-rwxr-xr-x  1 root root 11676672 May 15 19:07 org.clearlinux.native.5.17.8-1143
-rwxr-xr-x  1 root root 11676992 May 18 06:09 org.clearlinux.native.5.17.9-1144
-rwxr-xr-x  1 root root 11688928 May 20 10:46 org.clearlinux.native.5.17.9-1145
win@cnx~ $ sudo coredumpctl list
TIME                           PID  UID  GID SIG     COREFILE EXE                             SIZE
Thu 2022-05-19 14:35:09 EDT 139196    0    0 SIGSEGV missing  /usr/bin/clr-boot-manager        n/a
Mon 2022-05-23 15:25:28 EDT  36645    0    0 SIGSEGV present  /usr/bin/clr-boot-manager      36.5K
Wed 2022-05-25 16:26:04 EDT  91081    0    0 SIGSEGV none     /usr/bin/clr-boot-manager        n/a
Wed 2022-05-25 18:08:41 EDT  93980    0    0 SIGSEGV none     /usr/bin/clr-boot-manager        n/a
Wed 2022-05-25 18:10:41 EDT   1727    0    0 SIGSEGV none     /usr/bin/clr-boot-manager        n/a
Wed 2022-05-25 18:16:36 EDT   2308    0    0 SIGSEGV none     /usr/bin/clr-boot-manager        n/a
win@cnx~ $ sudo coredumpctl dump 36645 --output clr-boot-manager-coredump-2022-05-23
           PID: 36645 (clr-boot-manage)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Mon 2022-05-23 15:25:28 EDT (2 days ago)
  Command Line: /usr/bin/clr-boot-manager update
    Executable: /usr/bin/clr-boot-manager
 Control Group: /system.slice/swupd-update.service
          Unit: swupd-update.service
         Slice: system.slice
       Boot ID: eb528dc6328943e1bf4f70f8e18e217f
    Machine ID: cf058b305e544553bc82470db8c668bb
      Hostname: cnx
       Storage: /var/lib/systemd/coredump/core.clr-boot-manage.0.eb528dc6328943e1bf4f70f8e18e217f.36645.1653333928000000.zst (present)
     Disk Size: 36.5K
       Message: Process 36645 (clr-boot-manage) of user 0 dumped core.

                Found module linux-vdso.so.1 with build-id: 87503f7fa4e1bbaf7b0ca0e11195ca74212c2c9a
                Found module ld-linux-x86-64.so.2 with build-id: be5010234b1373e9a66a03b4a36f6be392cb2ce0
                Found module libdl.so.2 with build-id: e9d86106e31bef2ceea474ba248675a816f2c322
                Found module libc.so.6 with build-id: 307ac5841520c81a5456f7de4dfe31dae01377e7
                Found module libgcc_s.so.1 with build-id: c1b8926c6a0e527d40f378065915dc55429c5749
                Found module libefivar.so.1 with build-id: b3b3acd0a6b0bd24db7a1cd1c05146967cb731e1
                Found module libefiboot.so.1 with build-id: 89998560d05388a31eba3cea13f00df0a5ae2f22
                Found module libblkid.so.1 with build-id: 19535a075bbc6631601a7c12a17c89850d72b714
                Found module clr-boot-manager with build-id: f96221201beb401aaaf3405c3dad563ba9ef575d
                Stack trace of thread 36645:
                #0  0x00007f025c1c8d86 blkid_do_safeprobe (libblkid.so.1 + 0xbd86)
                #1  0x000055a2e161b6f6 get_parent_disk_devno (clr-boot-manager + 0xd6f6)
                #2  0x000055a2e161b9f3 get_legacy_boot_device (clr-boot-manager + 0xd9f3)
                #3  0x000055a2e16193d8 cmb_inspect_root_native (clr-boot-manager + 0xb3d8)
                #4  0x000055a2e16150c3 boot_manager_set_prefix (clr-boot-manager + 0x70c3)
                #5  0x000055a2e1614e02 cbm_command_update_do (clr-boot-manager + 0x6e02)
                #6  0x000055a2e1614edc cbm_command_update (clr-boot-manager + 0x6edc)
                #7  0x000055a2e1613772 main (clr-boot-manager + 0x5772)
                #8  0x00007f025bf5164e n/a (libc.so.6 + 0x2964e)
                #9  0x00007f025bf51709 __libc_start_main (libc.so.6 + 0x29709)
                #10 0x000055a2e1613945 _start (clr-boot-manager + 0x5945)
win@cnx~ $ 
nanon@hkg~ $ clr-network-troubleshooter
clr-network-troubleshooter v15
Running on Clear Linux release 36400
---------------------------------

Testing https site: [curl --connect-timeout 10 -s -i -L https://download.clearlinux.org/latest]
  ✓ Got release 36400

Testing http site: [curl --connect-timeout 10 -s -i -L http://neverssl.com/]
  ✓ OK

=====================
Connectivity seems ok
nanon@hkg~ $ sudo clr-boot-manager list-kernels
Password: 
Segmentation fault
nanon@hkg~ $ uname -r
5.13.13-1070.native
nanon@hkg~ $ ls -al /usr/lib/kernel
total 46068
drwxr-xr-x  3 root root     4096 May 23 15:58 .
drwxr-xr-x 19 root root     4096 May 25 17:13 ..
-rw-r--r--  2 root root      267 Aug 26  2021 cmdline-5.13.13-1070.native
-rw-r--r--  2 root root      267 Aug 26  2021 cmdline-5.17.8-1143.native
-rw-r--r--  1 root root      267 May 18 06:09 cmdline-5.17.9-1144.native
-rw-r--r--  1 root root      267 May 23 15:57 cmdline-5.17.9-1145.native
-rw-r--r--  1 root root   227314 Aug 26  2021 config-5.13.13-1070.native
-rw-r--r--  1 root root   234474 May 15 19:07 config-5.17.8-1143.native
-rw-r--r--  1 root root   234474 May 18 06:09 config-5.17.9-1144.native
-rw-r--r--  1 root root   234463 May 23 15:57 config-5.17.9-1145.native
lrwxrwxrwx  1 root root       33 May 20 10:48 default-native -> org.clearlinux.native.5.17.9-1145
drwxr-xr-x  2 root root     4096 May 19 10:00 install.d
-rwxr-xr-x  1 root root 11154624 Aug 26  2021 org.clearlinux.native.5.13.13-1070
-rwxr-xr-x  1 root root 11676672 May 15 19:07 org.clearlinux.native.5.17.8-1143
-rwxr-xr-x  1 root root 11676992 May 18 06:09 org.clearlinux.native.5.17.9-1144
-rwxr-xr-x  1 root root 11688928 May 20 10:46 org.clearlinux.native.5.17.9-1145
nanon@hkg~ $ sudo coredumpctl list
TIME                         PID UID GID SIG     COREFILE EXE                        SIZE
Thu 2022-05-19 14:31:36 EDT 4088   0   0 SIGSEGV missing  /usr/bin/clr-boot-manager   n/a
Thu 2022-05-19 15:57:39 EDT  450   0   0 SIGSEGV none     /usr/bin/clr-boot-manager   n/a
Mon 2022-05-23 15:58:31 EDT 5900   0   0 SIGSEGV present  /usr/bin/clr-boot-manager 36.3K
Wed 2022-05-25 14:47:12 EDT  527   0   0 SIGSEGV none     /usr/bin/clr-boot-manager   n/a
Wed 2022-05-25 14:49:48 EDT  919   0   0 SIGSEGV none     /usr/bin/clr-boot-manager   n/a
Wed 2022-05-25 16:27:21 EDT 1600   0   0 SIGSEGV none     /usr/bin/clr-boot-manager   n/a
Wed 2022-05-25 16:50:33 EDT 1989   0   0 SIGSEGV none     /usr/bin/clr-boot-manager   n/a
Wed 2022-05-25 18:14:49 EDT  407   0   0 SIGSEGV none     /usr/bin/clr-boot-manager   n/a
Wed 2022-05-25 18:19:40 EDT  462   0   0 SIGSEGV none     /usr/bin/clr-boot-manager   n/a
Wed 2022-05-25 18:32:23 EDT  526   0   0 SIGSEGV none     /usr/bin/clr-boot-manager   n/a
nanon@hkg~ $ sudo coredumpctl dump 5900 --output clr-boot-manager-coredump-2022-05-23
           PID: 5900 (clr-boot-manage)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Mon 2022-05-23 15:58:30 EDT (2 days ago)
  Command Line: /usr/bin/clr-boot-manager update
    Executable: /usr/bin/clr-boot-manager
 Control Group: /system.slice/swupd-update.service
          Unit: swupd-update.service
         Slice: system.slice
       Boot ID: 38a23a5f419c4a0cbce48d9c16e0004e
    Machine ID: 23d2dee1360b4c088da36d9d5ec701a4
      Hostname: hkg
       Storage: /var/lib/systemd/coredump/core.clr-boot-manage.0.38a23a5f419c4a0cbce48d9c16e0004e.5900.1653335910000000.zst (present)
     Disk Size: 36.3K
       Message: Process 5900 (clr-boot-manage) of user 0 dumped core.

                Found module linux-vdso.so.1 with build-id: bbb563fe520f964d276fef5b9b58cb9598dac95c
                Found module ld-linux-x86-64.so.2 with build-id: be5010234b1373e9a66a03b4a36f6be392cb2ce0
                Found module libdl.so.2 with build-id: e9d86106e31bef2ceea474ba248675a816f2c322
                Found module libc.so.6 with build-id: 77a4b40b67db7f6ceee489f84e75f4540bd6bd1c
                Found module libgcc_s.so.1 with build-id: 8f32ee7bffaeb40505d2ee88727e05a39d2e9534
                Found module libefivar.so.1 with build-id: b3b3acd0a6b0bd24db7a1cd1c05146967cb731e1
                Found module libefiboot.so.1 with build-id: 89998560d05388a31eba3cea13f00df0a5ae2f22
                Found module libblkid.so.1 with build-id: 19535a075bbc6631601a7c12a17c89850d72b714
                Found module clr-boot-manager with build-id: f96221201beb401aaaf3405c3dad563ba9ef575d
                Stack trace of thread 5900:
                #0  0x00007f7bc4261d86 blkid_do_safeprobe (libblkid.so.1 + 0xbd86)
                #1  0x000055da137de6f6 n/a (clr-boot-manager + 0xd6f6)
                #2  0x000055da137de9f3 n/a (clr-boot-manager + 0xd9f3)
                #3  0x000055da137dc3d8 n/a (clr-boot-manager + 0xb3d8)
                #4  0x000055da137d80c3 n/a (clr-boot-manager + 0x70c3)
                #5  0x000055da137d7e02 n/a (clr-boot-manager + 0x6e02)
                #6  0x000055da137d7edc n/a (clr-boot-manager + 0x6edc)
                #7  0x000055da137d6772 n/a (clr-boot-manager + 0x5772)
                #8  0x00007f7bc3feb4de n/a (libc.so.6 + 0x294de)
                #9  0x00007f7bc3feb5a9 __libc_start_main (libc.so.6 + 0x295a9)
                #10 0x000055da137d6945 n/a (clr-boot-manager + 0x5945)
nanon@hkg~ $ 
nnn@enf~ $ clr-network-troubleshooter
clr-network-troubleshooter v15
Running on Clear Linux release 36400
---------------------------------

Testing https site: [curl --connect-timeout 10 -s -i -L https://download.clearlinux.org/latest]
  ✓ Got release 36400

Testing http site: [curl --connect-timeout 10 -s -i -L http://neverssl.com/]
  ✓ OK

=====================
Connectivity seems ok
nnn@enf~ $ sudo clr-boot-manager list-kernels
Password: 
Segmentation fault
nnn@enf~ $ uname -r
5.13.19-547.kvm
nnn@enf~ $ ls -al /usr/lib/kernel
total 46528
drwxr-xr-x  3 root root     4096 May 24 15:33 .
drwxr-xr-x 18 root root     4096 May 24 22:19 ..
-rw-r--r--  1 root root      220 Sep 18  2021 cmdline-5.13.19-547.kvm
-rw-r--r--  1 root root      267 Feb 17 21:53 cmdline-5.16.10-1127.native
-rw-r--r--  1 root root      267 Mar 17 16:17 cmdline-5.16.15-1134.native
-rw-r--r--  1 root root      267 May 20 10:46 cmdline-5.17.9-1145.native
-rw-r--r--  1 root root   124550 Sep 18  2021 config-5.13.19-547.kvm
-rw-r--r--  1 root root   232436 Feb 17 21:53 config-5.16.10-1127.native
-rw-r--r--  1 root root   232392 Mar 17 16:17 config-5.16.15-1134.native
-rw-r--r--  1 root root   234463 May 20 10:46 config-5.17.9-1145.native
lrwxrwxrwx  1 root root       30 Sep 18  2021 default-kvm -> org.clearlinux.kvm.5.13.19-547
lrwxrwxrwx  1 root root       33 May 20 10:48 default-native -> org.clearlinux.native.5.17.9-1145
drwxr-xr-x  2 root root     4096 Feb 16 20:28 install.d
-rwxr-xr-x  1 root root 11885440 Sep 18  2021 org.clearlinux.kvm.5.13.19-547
-rwxr-xr-x  1 root root 11592608 Feb 16 12:05 org.clearlinux.native.5.16.10-1127
-rwxr-xr-x  1 root root 11611872 Mar 16 10:07 org.clearlinux.native.5.16.15-1134
-rwxr-xr-x  1 root root 11688928 May 20 10:46 org.clearlinux.native.5.17.9-1145
nnn@enf~ $ sudo coredumpctl list
TIME                         PID UID GID SIG     COREFILE EXE                        SIZE
Tue 2022-05-24 15:33:49 EDT 9754   0   0 SIGSEGV present  /usr/bin/clr-boot-manager 36.6K
Tue 2022-05-24 16:04:25 EDT  231   0   0 SIGSEGV none     /usr/bin/clr-boot-manager   n/a
Tue 2022-05-24 19:03:16 EDT  369   0   0 SIGSEGV none     /usr/bin/clr-boot-manager   n/a
Wed 2022-05-25 18:41:06 EDT 1228   0   0 SIGSEGV none     /usr/bin/clr-boot-manager   n/a
nnn@enf~ $ sudo coredumpctl dump 9754 --output clr-boot-manager-coredump-2022-05-24
           PID: 9754 (clr-boot-manage)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Tue 2022-05-24 15:33:49 EDT (1 day 3h ago)
  Command Line: /usr/bin/clr-boot-manager update
    Executable: /usr/bin/clr-boot-manager
 Control Group: /system.slice/swupd-update.service
          Unit: swupd-update.service
         Slice: system.slice
       Boot ID: 9f04f8da8eb5425ba7c77b15f35ace60
    Machine ID: 00501d3165af4fafb09243b77a752288
      Hostname: www
       Storage: /var/lib/systemd/coredump/core.clr-boot-manage.0.9f04f8da8eb5425ba7c77b15f35ace60.9754.1653420829000000.zst (present)
     Disk Size: 36.6K
       Message: Process 9754 (clr-boot-manage) of user 0 dumped core.

                Found module linux-vdso.so.1 with build-id: ece4b258ae84300fec9404671c85c63c5db0f4a9
                Found module ld-linux-x86-64.so.2 with build-id: be5010234b1373e9a66a03b4a36f6be392cb2ce0
                Found module libdl.so.2 with build-id: e9d86106e31bef2ceea474ba248675a816f2c322
                Found module libc.so.6 with build-id: 77a4b40b67db7f6ceee489f84e75f4540bd6bd1c
                Found module libgcc_s.so.1 with build-id: 8f32ee7bffaeb40505d2ee88727e05a39d2e9534
                Found module libefivar.so.1 with build-id: b3b3acd0a6b0bd24db7a1cd1c05146967cb731e1
                Found module libefiboot.so.1 with build-id: 89998560d05388a31eba3cea13f00df0a5ae2f22
                Found module libblkid.so.1 with build-id: 19535a075bbc6631601a7c12a17c89850d72b714
                Found module clr-boot-manager with build-id: f96221201beb401aaaf3405c3dad563ba9ef575d
                Stack trace of thread 9754:
                #0  0x00007faeb81aed86 blkid_do_safeprobe (libblkid.so.1 + 0xbd86)
                #1  0x0000564ba7c736f6 n/a (clr-boot-manager + 0xd6f6)
                #2  0x0000564ba7c739f3 n/a (clr-boot-manager + 0xd9f3)
                #3  0x0000564ba7c713d8 n/a (clr-boot-manager + 0xb3d8)
                #4  0x0000564ba7c6d0c3 n/a (clr-boot-manager + 0x70c3)
                #5  0x0000564ba7c6ce02 n/a (clr-boot-manager + 0x6e02)
                #6  0x0000564ba7c6cedc n/a (clr-boot-manager + 0x6edc)
                #7  0x0000564ba7c6b772 n/a (clr-boot-manager + 0x5772)
                #8  0x00007faeb7f384de n/a (libc.so.6 + 0x294de)
                #9  0x00007faeb7f385a9 __libc_start_main (libc.so.6 + 0x295a9)
                #10 0x0000564ba7c6b945 n/a (clr-boot-manager + 0x5945)
nnn@enf~ $
bryteise commented 2 years ago

Hrm given the lack of cleanup I wonder if it has been failing since August for you or is that just the case of you updating the machine after a long period of keeping the same version?

Regardless the backtrace from win is certainly helpful as a place to start. I haven't yet seen this behavior myself so this might be a bit.

thiagomacieira commented 2 years ago

Core dump with more debug symbols:

(gdb) bt
#0  0x00007faeb81aed86 in blkid_do_safeprobe (pr=0x0) at libblkid/src/probe.c:1561
#1  0x0000564ba7c736f6 in get_parent_disk_devno (diskdevno=<synthetic pointer>, path=0x564ba92f3520 "/") at ../src/lib/files.c:166
#2  get_parent_disk (path=0x564ba92f3520 "/") at ../src/lib/files.c:208
#3  0x0000564ba7c739f3 in get_legacy_boot_device (path=path@entry=0x564ba92f3520 "/") at ../src/lib/files.c:283
#4  0x0000564ba7c713d8 in cmb_inspect_root_native (realp=0x564ba92f3520 "/", c=0x564ba92f46a0) at ../src/bootman/sysconfig.c:183
#5  cbm_inspect_root (path=path@entry=0x564ba7c80931 "/", image_mode=<optimized out>) at ../src/bootman/sysconfig.c:243
#6  0x0000564ba7c6d0c3 in boot_manager_set_prefix (self=self@entry=0x564ba92f3e50, prefix=prefix@entry=0x564ba7c80931 "/") at ../src/bootman/bootman.c:184
#7  0x0000564ba7c6ce02 in cbm_command_update_do (manager=0x564ba92f3e50, root=0x0, forced_image=<optimized out>) at ../src/cli/ops/update.c:77
#8  0x0000564ba7c6cedc in cbm_command_update (argc=<optimized out>, argv=<optimized out>) at ../src/cli/ops/update.c:44
#9  0x0000564ba7c6b772 in main (argc=0, argv=0x7ffe36b4db68) at ../src/cli/main.c:280

It's a null-pointer dereference inside libblkid, called from https://github.com/clearlinux/clr-boot-manager/blob/master/src/lib/files.c#L166. In frame 1, dev_path is also null, which probably means cbm_system_get_device_for_mountpoint("/") returned null.

Can you share the output of lsblk in your system?

cphuntington97 commented 2 years ago

Hrm given the lack of cleanup I wonder if it has been failing since August for you or is that just the case of you updating the machine after a long period of keeping the same version?

win@cnx is my year-old bare metal desktop. Auto updates have been on this entire time.

nanon@hkg is a week old bare metal server iso install; I just checked the flash drive I used and it has 5.13.13-1070 kernel on it

nnn@enf is a KVM guest on UpCloud. I installed a pretty old server iso (35000) in February. I added the kernel-kvm bundle, but didn't realize I needed to switch to the kvm kernel with clr-boot-manager until yesterday. I experienced a lot of instability trying to use the native kernel as a KVM guest, so it has been up and down during this time.

cphuntington97 commented 2 years ago

Can you share the output of lsblk in your system?

win@cnx~ $ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
sda           8:0    0   3.6T  0 disk  
└─sda       252:1    0   3.6T  0 crypt /home/win/disks/btrfs-array
sdb           8:16   0   3.6T  0 disk  
└─sdb       252:0    0   3.6T  0 crypt 
sdc           8:32   0   3.6T  0 disk  
└─sdc       252:3    0   3.6T  0 crypt 
sdd           8:48   0   3.6T  0 disk  
└─sdd       252:2    0   3.6T  0 crypt 
sde           8:64   0 931.5G  0 disk  
└─sde1        8:65   0 931.5G  0 part  /home/win/disks/ssd
sdf           8:80   1     0B  0 disk  
sdg           8:96   1     0B  0 disk  
sdh           8:112  1     0B  0 disk  
sdi           8:128  1  14.6G  0 disk  
├─sdi1        8:129  1   228M  0 part  /run/media/win/pmOS_boot
└─sdi2        8:130  1   1.6G  0 part  /run/media/win/pmOS_root
sdj           8:144  1  29.2G  0 disk  
└─sdj1        8:145  1  29.2G  0 part  /run/media/win/EOS_DIGITAL
sdk           8:160  1  28.9G  0 disk  
├─sdk1        8:161  1     1G  0 part  /run/media/win/CLR_ISO
└─sdk2        8:162  1   100M  0 part  
nvme0n1     259:0    0 465.8G  0 disk  
├─nvme0n1p1 259:1    0   149M  0 part  
└─nvme0n1p2 259:2    0 465.6G  0 part  /
win@cnx~ $ 
nanon@hkg~ $ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda      8:0    0 465.8G  0 disk 
├─sda1   8:1    0   149M  0 part 
└─sda2   8:2    0 465.6G  0 part /
nanon@hkg~ $ 
nnn@enf~ $ lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
vda    254:0    0   25G  0 disk 
├─vda1 254:1    0  149M  0 part 
└─vda2 254:2    0 24.9G  0 part /
nnn@enf~ $
bryteise commented 2 years ago

Are the systems all EFI? It looks like there is something going wrong with probing for the rootfs disk after poking the backtrace. What do you get from cat /proc/self/mounts on these systems?

cphuntington97 commented 2 years ago

Are the systems all EFI? It looks like there is something going wrong with probing for the rootfs disk after poking the backtrace. What do you get from cat /proc/self/mounts on these systems?

win@cnx~ $ cat /proc/self/mounts
/dev/root / ext4 rw,relatime,stripe=256 0 0
devtmpfs /dev devtmpfs rw,relatime,size=8072692k,nr_inodes=2018173,mode=755,inode64 0 0
/proc /proc proc rw,relatime 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev,noexec,inode64 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,nodev,size=3231360k,nr_inodes=819200,mode=755,inode64 0 0
tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,size=4096k,nr_inodes=1024,mode=755,inode64 0 0
cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0
cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0
pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,nosuid,nodev,noexec,relatime,pagesize=2M 0 0
mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0
fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0
configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0
debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
cgroup /sys/fs/cgroup/misc cgroup rw,nosuid,nodev,noexec,relatime,misc 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16321 0 0
tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /tmp tmpfs rw,nosuid,nodev,size=8078396k,nr_inodes=1048576,inode64 0 0
/dev/sde1 /home/win/disks/ssd ext4 rw,relatime,stripe=256 0 0
/dev/mapper/sda /home/win/disks/btrfs-array btrfs rw,relatime,compress=lzo,space_cache,subvolid=258,subvol=/data 0 0
clr_debug_fuse /usr/src/debug fuse.clr_debug_fuse rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other 0 0
clr_debug_fuse /usr/lib/debug fuse.clr_debug_fuse rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other 0 0
tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=1615676k,nr_inodes=403919,mode=700,uid=1000,gid=1000,inode64 0 0
gvfsd-fuse /run/user/1000/gvfs fuse.gvfsd-fuse rw,nosuid,nodev,relatime,user_id=1000,group_id=1000 0 0
/dev/sdj1 /run/media/win/EOS_DIGITAL vfat rw,nosuid,nodev,relatime,uid=1000,gid=1000,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,showexec,utf8,flush,errors=remount-ro 0 0
/dev/sdi1 /run/media/win/pmOS_boot ext2 ro,nosuid,nodev,relatime,errors=remount-ro 0 0
/dev/sdi2 /run/media/win/pmOS_root ext4 rw,nosuid,nodev,relatime,errors=remount-ro 0 0
portal /run/user/1000/doc fuse.portal rw,nosuid,nodev,relatime,user_id=1000,group_id=1000 0 0
/dev/sdk1 /run/media/win/CLR_ISO iso9660 ro,nosuid,nodev,relatime,nojoliet,check=s,map=n,blocksize=2048,uid=1000,gid=1000,dmode=500,fmode=400,iocharset=utf8 0 0
win@cnx~ $ 
nanon@hkg~ $ cat /proc/self/mounts
/dev/root / ext4 rw,relatime,stripe=256 0 0
devtmpfs /dev devtmpfs rw,relatime,size=4019204k,nr_inodes=1004801,mode=755,inode64 0 0
/proc /proc proc rw,relatime 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev,noexec,inode64 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,nodev,size=1610056k,nr_inodes=819200,mode=755,inode64 0 0
tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,size=4096k,nr_inodes=1024,mode=755,inode64 0 0
cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0
cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0
pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,nosuid,nodev,noexec,relatime,pagesize=2M 0 0
mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0
fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0
configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0
debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
cgroup /sys/fs/cgroup/misc cgroup rw,nosuid,nodev,noexec,relatime,misc 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15231 0 0
tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /tmp tmpfs rw,nosuid,nodev,nr_inodes=1048576,inode64 0 0
clr_debug_fuse /usr/src/debug fuse.clr_debug_fuse rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other 0 0
clr_debug_fuse /usr/lib/debug fuse.clr_debug_fuse rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other 0 0
tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=805028k,nr_inodes=201257,mode=700,uid=1000,gid=1000,inode64 0 0
nanon@hkg~ $ 
nnn@enf~ $ cat /proc/self/mounts
/dev/root / ext4 rw,relatime,stripe=256 0 0
devtmpfs /dev devtmpfs rw,relatime,size=491648k,nr_inodes=122912,mode=755,inode64 0 0
/proc /proc proc rw,relatime 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev,noexec,inode64 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,nodev,size=197396k,nr_inodes=819200,mode=755,inode64 0 0
tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,size=4096k,nr_inodes=1024,mode=755,inode64 0 0
cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0
cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0
pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,nosuid,nodev,noexec,relatime,pagesize=2M 0 0
mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0
debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0
cgroup /sys/fs/cgroup/misc cgroup rw,nosuid,nodev,noexec,relatime,misc 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6157 0 0
tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /tmp tmpfs rw,nosuid,nodev,nr_inodes=1048576,inode64 0 0
nnn@enf~ $
cphuntington97 commented 2 years ago 
win@cnx~ $ systemctl status boot.mount
○ boot.mount - EFI System Partition Automount
     Loaded: loaded (/run/systemd/generator.late/boot.mount; generated)
     Active: inactive (dead)
      Where: /boot
       What: /dev/nvme0n1p1
       Docs: man:systemd-gpt-auto-generator(8)
win@cnx~ $
nanon@hkg~ $ systemctl status boot.mount
○ boot.mount - EFI System Partition Automount
     Loaded: loaded (/run/systemd/generator.late/boot.mount; generated)
     Active: inactive (dead)
      Where: /boot
       What: /dev/sda1
       Docs: man:systemd-gpt-auto-generator(8)
nanon@hkg~ $ 
nnn@enf~ $ systemctl status boot.mount
○ boot.mount - EFI System Partition Automount
     Loaded: loaded (/run/systemd/generator.late/boot.mount; generated)
     Active: inactive (dead)
      Where: /boot
       What: /dev/vda1
       Docs: man:systemd-gpt-auto-generator(8)
nnn@enf~ $
cphuntington97 commented 2 years ago 
win@cnx~ $ lsblk -o 'NAME,MAJ:MIN,RM,SIZE,RO,PARTTYPENAME,TYPE,PARTLABEL,MOUNTPOINTS'
NAME        MAJ:MIN RM   SIZE RO PARTTYPENAME        TYPE  PARTLABEL MOUNTPOINTS
sda           8:0    0   3.6T  0                     disk            
└─sda       252:1    0   3.6T  0                     crypt           /home/win/disks/btrfs-array
sdb           8:16   0   3.6T  0                     disk            
└─sdb       252:0    0   3.6T  0                     crypt           
sdc           8:32   0   3.6T  0                     disk            
└─sdc       252:3    0   3.6T  0                     crypt           
sdd           8:48   0   3.6T  0                     disk            
└─sdd       252:2    0   3.6T  0                     crypt           
sde           8:64   0 931.5G  0                     disk            
└─sde1        8:65   0 931.5G  0 Linux filesystem    part            /home/win/disks/ssd
sdf           8:80   1     0B  0                     disk            
sdg           8:96   1     0B  0                     disk            
sdh           8:112  1     0B  0                     disk            
sdi           8:128  1  14.6G  0                     disk            
├─sdi1        8:129  1   228M  0 Linux               part            /run/media/win/pmOS_boot
└─sdi2        8:130  1   1.6G  0 Linux               part            /run/media/win/pmOS_root
sdj           8:144  1  29.2G  0                     disk            
└─sdj1        8:145  1  29.2G  0 W95 FAT32 (LBA)     part            /run/media/win/EOS_DIGITAL
sdk           8:160  1  28.9G  0                     disk            
├─sdk1        8:161  1     1G  0 Empty               part            /run/media/win/CLR_ISO
└─sdk2        8:162  1   100M  0 EFI (FAT-12/16/32)  part            
nvme0n1     259:0    0 465.8G  0                     disk            
├─nvme0n1p1 259:1    0   149M  0 EFI System          part  EFI       
└─nvme0n1p2 259:2    0 465.6G  0 Linux root (x86-64) part  /         /
win@cnx~ $ 
nanon@hkg~ $ lsblk -o 'NAME,MAJ:MIN,RM,SIZE,RO,PARTTYPENAME,TYPE,PARTLABEL,MOUNTPOINTS'
NAME   MAJ:MIN RM   SIZE RO PARTTYPENAME        TYPE PARTLABEL MOUNTPOINTS
sda      8:0    0 465.8G  0                     disk           
├─sda1   8:1    0   149M  0 EFI System          part EFI       
└─sda2   8:2    0 465.6G  0 Linux root (x86-64) part /         /
nanon@hkg~ $ 
nnn@enf~ $ lsblk -o 'NAME,MAJ:MIN,RM,SIZE,RO,PARTTYPENAME,TYPE,PARTLABEL,MOUNTPOINTS'
NAME   MAJ:MIN RM  SIZE RO PARTTYPENAME        TYPE PARTLABEL MOUNTPOINTS
vda    254:0    0   25G  0                     disk           
├─vda1 254:1    0  149M  0 EFI System          part EFI       
└─vda2 254:2    0 24.9G  0 Linux root (x86-64) part /         /
nnn@enf~ $
bryteise commented 2 years ago

Well that all looks normal and in theory I have a patch for this that should have fixed it but it appears to not be working in your case. What do you get from ls -l /dev/gpt-auto-root? If that isn't there then that's the problem but it should be given you have the PARTTYPE set (based on the PARTTYPENAME information).

cphuntington97 commented 2 years ago

What do you get from ls -l /dev/gpt-auto-root?

win@cnx~ $ ls -l /dev/gpt-auto-root
ls: cannot access '/dev/gpt-auto-root': No such file or directory
win@cnx~ $ 
nanon@hkg~ $ ls -l /dev/gpt-auto-root
ls: cannot access '/dev/gpt-auto-root': No such file or directory
nanon@hkg~ $
nnn@enf~ $ ls -l /dev/gpt-auto-root
ls: cannot access '/dev/gpt-auto-root': No such file or directory
nnn@enf~ $
cphuntington97 commented 2 years ago

Are the systems all EFI?

I actually don't think any of them are, if this is a reliable test.

win@cnx~ $ sudo efibootmgr
EFI variables are not supported on this system.
win@cnx~ $ 
nanon@hkg~ $ sudo efibootmgr
EFI variables are not supported on this system.
nanon@hkg~ $ 
nnn@enf~ $ sudo efibootmgr
EFI variables are not supported on this system.
nnn@enf~ $
bryteise commented 2 years ago

Alrighty, udev not doing what I expect. Is this using luks?

Anyway lets see what udev says it is doing, could you try a sudo udevadm test /sys/block/DEV/DEV# Where DEV is sda on hkg and DEV# is the root partiton so sda2 on hkg.

bryteise commented 2 years ago

Are the systems all EFI?

I actually don't think any of them are, if this is a reliable test.

Uh what is in /boot (both before an after mounting the EFI partition)? This installation seems a little odd.

cphuntington97 commented 2 years ago

Alrighty, udev not doing what I expect. Is this using luks?

there are some luks disks on cnx but none of the system drives

Anyway lets see what udev says it is doing, could you try a sudo udevadm test /sys/block/DEV/DEV# Where DEV is sda on hkg and DEV# is the root partiton so sda2 on hkg.

nanon@hkg~ $ sudo udevadm test /sys/block/sda/sda2
Password: 
This program is for debugging only, it does not run any program
specified by a RUN key. It may show incorrect results, because
some values may be different, or not available at a simulation run.

Trying to open "/etc/systemd/hwdb/hwdb.bin"...
Trying to open "/etc/udev/hwdb.bin"...
Trying to open "/usr/lib/systemd/hwdb/hwdb.bin"...
Trying to open "/usr/lib/udev/hwdb.bin"...
=== trie on-disk ===
tool version:          249
file size:          200150 bytes
header size             80 bytes
strings              43030 bytes
nodes               157040 bytes
Load module index
Found cgroup2 on /sys/fs/cgroup/unified, unified hierarchy for systemd controller
Found container virtualization none.
Loaded timestamp for '/run/systemd/network'.
Parsed configuration file /usr/lib/systemd/network/99-default.link
Created link configuration context.
Loaded timestamp for '/usr/lib/udev/rules.d'.
Reading rules file: /usr/lib/udev/rules.d/50-udev-default.rules
Reading rules file: /usr/lib/udev/rules.d/60-autosuspend.rules
Reading rules file: /usr/lib/udev/rules.d/60-block.rules
Reading rules file: /usr/lib/udev/rules.d/60-drm.rules
Reading rules file: /usr/lib/udev/rules.d/60-evdev.rules
Reading rules file: /usr/lib/udev/rules.d/60-fido-id.rules
Reading rules file: /usr/lib/udev/rules.d/60-input-id.rules
Reading rules file: /usr/lib/udev/rules.d/60-persistent-input.rules
Reading rules file: /usr/lib/udev/rules.d/60-persistent-storage.rules
Reading rules file: /usr/lib/udev/rules.d/60-rdma-ndd.rules
Reading rules file: /usr/lib/udev/rules.d/60-rdma-persistent-naming.rules
Reading rules file: /usr/lib/udev/rules.d/60-sensor.rules
Reading rules file: /usr/lib/udev/rules.d/60-serial.rules
Reading rules file: /usr/lib/udev/rules.d/60-srp_daemon.rules
Reading rules file: /usr/lib/udev/rules.d/60-tbtacl.rules
Reading rules file: /usr/lib/udev/rules.d/60-tbtxdomain.rules
Reading rules file: /usr/lib/udev/rules.d/64-btrfs-dm.rules
Reading rules file: /usr/lib/udev/rules.d/64-btrfs-zoned.rules
Reading rules file: /usr/lib/udev/rules.d/64-btrfs.rules
Reading rules file: /usr/lib/udev/rules.d/70-camera.rules
Reading rules file: /usr/lib/udev/rules.d/70-memory.rules
Reading rules file: /usr/lib/udev/rules.d/70-mouse.rules
Reading rules file: /usr/lib/udev/rules.d/70-power-switch.rules
Reading rules file: /usr/lib/udev/rules.d/70-touchpad.rules
Reading rules file: /usr/lib/udev/rules.d/70-uaccess.rules
Reading rules file: /usr/lib/udev/rules.d/71-seat.rules
Reading rules file: /usr/lib/udev/rules.d/73-seat-late.rules
Reading rules file: /usr/lib/udev/rules.d/75-net-description.rules
Reading rules file: /usr/lib/udev/rules.d/75-rdma-description.rules
Reading rules file: /usr/lib/udev/rules.d/80-drivers.rules
Reading rules file: /usr/lib/udev/rules.d/80-kvm.rules
Reading rules file: /usr/lib/udev/rules.d/80-net-setup-link.rules
Reading rules file: /usr/lib/udev/rules.d/81-net-dhcp.rules
Reading rules file: /usr/lib/udev/rules.d/90-iwpmd.rules
Reading rules file: /usr/lib/udev/rules.d/90-rdma-hw-modules.rules
Reading rules file: /usr/lib/udev/rules.d/90-rdma-ulp-modules.rules
Reading rules file: /usr/lib/udev/rules.d/90-rdma-umad.rules
Reading rules file: /usr/lib/udev/rules.d/90-tpm-thinkpad.rules
Reading rules file: /usr/lib/udev/rules.d/90-vconsole.rules
Reading rules file: /usr/lib/udev/rules.d/99-systemd.rules
sda2: /usr/lib/udev/rules.d/50-udev-default.rules:69 GROUP 6
sda2: /usr/lib/udev/rules.d/60-persistent-storage.rules:69 LINK 'disk/by-id/ata-Samsung_SSD_860_EVO_500GB_S3Z1NB0K102757R-part2'
sda2: /usr/lib/udev/rules.d/60-persistent-storage.rules:93 LINK 'disk/by-path/pci-0000:00:13.0-ata-1.0-part2'
sda2: /usr/lib/udev/rules.d/60-persistent-storage.rules:96 LINK 'disk/by-path/pci-0000:00:13.0-ata-1-part2'
sda2: /usr/lib/udev/rules.d/60-persistent-storage.rules:110 Importing properties from results of builtin command 'blkid'
sda2: Probe /dev/sda2 with raid and offset=0
sda2: /usr/lib/udev/rules.d/60-persistent-storage.rules:113 LINK 'disk/by-uuid/b0828b38-a6c0-4d3f-abb9-b6e98bc378be'
sda2: /usr/lib/udev/rules.d/60-persistent-storage.rules:114 LINK 'disk/by-label/root'
sda2: /usr/lib/udev/rules.d/60-persistent-storage.rules:118 LINK 'disk/by-id/wwn-0x5002538e40060f08-part2'
sda2: /usr/lib/udev/rules.d/60-persistent-storage.rules:121 LINK 'disk/by-partuuid/42632089-53bc-48f5-bded-3c98ea77af8f'
sda2: /usr/lib/udev/rules.d/60-persistent-storage.rules:122 LINK 'disk/by-partlabel/\x2f'
sda2: Handling device node '/dev/sda2', devnum=b8:2
sda2: Preserve permissions of /dev/sda2, uid=0, gid=6, mode=0660
sda2: Found 'b8:2' claiming '/run/udev/links/disk\x2fby-partuuid\x2f42632089-53bc-48f5-bded-3c98ea77af8f'
sda2: Preserve already existing symlink '/dev/disk/by-partuuid/42632089-53bc-48f5-bded-3c98ea77af8f' to '../../sda2'
sda2: Found 'b8:2' claiming '/run/udev/links/disk\x2fby-path\x2fpci-0000:00:13.0-ata-1.0-part2'
sda2: Preserve already existing symlink '/dev/disk/by-path/pci-0000:00:13.0-ata-1.0-part2' to '../../sda2'
sda2: Found 'b8:2' claiming '/run/udev/links/disk\x2fby-id\x2fwwn-0x5002538e40060f08-part2'
sda2: Preserve already existing symlink '/dev/disk/by-id/wwn-0x5002538e40060f08-part2' to '../../sda2'
sda2: Found 'b8:2' claiming '/run/udev/links/disk\x2fby-partlabel\x2f\x5cx2f'
sda2: Preserve already existing symlink '/dev/disk/by-partlabel/\x2f' to '../../sda2'
sda2: Found 'b8:2' claiming '/run/udev/links/disk\x2fby-path\x2fpci-0000:00:13.0-ata-1-part2'
sda2: Preserve already existing symlink '/dev/disk/by-path/pci-0000:00:13.0-ata-1-part2' to '../../sda2'
sda2: Found 'b8:2' claiming '/run/udev/links/disk\x2fby-label\x2froot'
sda2: Preserve already existing symlink '/dev/disk/by-label/root' to '../../sda2'
sda2: Found 'b8:2' claiming '/run/udev/links/disk\x2fby-uuid\x2fb0828b38-a6c0-4d3f-abb9-b6e98bc378be'
sda2: Preserve already existing symlink '/dev/disk/by-uuid/b0828b38-a6c0-4d3f-abb9-b6e98bc378be' to '../../sda2'
sda2: Found 'b8:2' claiming '/run/udev/links/disk\x2fby-id\x2fata-Samsung_SSD_860_EVO_500GB_S3Z1NB0K102757R-part2'
sda2: Preserve already existing symlink '/dev/disk/by-id/ata-Samsung_SSD_860_EVO_500GB_S3Z1NB0K102757R-part2' to '../../sda2'
sda2: Preserve already existing symlink '/dev/block/8:2' to '../sda2'
sda2: sd-device: Created db file '/run/udev/data/b8:2' for '/devices/pci0000:00/0000:00:13.0/ata1/host0/target0:0:0/0:0:0:0/block/sda/sda2'
sda2: Handling device node '/dev/sda2', devnum=b8:2
sda2: Preserve permissions of /dev/sda2, uid=0, gid=6, mode=0660
sda2: Found 'b8:2' claiming '/run/udev/links/disk\x2fby-partuuid\x2f42632089-53bc-48f5-bded-3c98ea77af8f'
sda2: Preserve already existing symlink '/dev/disk/by-partuuid/42632089-53bc-48f5-bded-3c98ea77af8f' to '../../sda2'
sda2: Found 'b8:2' claiming '/run/udev/links/disk\x2fby-path\x2fpci-0000:00:13.0-ata-1.0-part2'
sda2: Preserve already existing symlink '/dev/disk/by-path/pci-0000:00:13.0-ata-1.0-part2' to '../../sda2'
sda2: Found 'b8:2' claiming '/run/udev/links/disk\x2fby-id\x2fwwn-0x5002538e40060f08-part2'
sda2: Preserve already existing symlink '/dev/disk/by-id/wwn-0x5002538e40060f08-part2' to '../../sda2'
sda2: Found 'b8:2' claiming '/run/udev/links/disk\x2fby-partlabel\x2f\x5cx2f'
sda2: Preserve already existing symlink '/dev/disk/by-partlabel/\x2f' to '../../sda2'
sda2: Found 'b8:2' claiming '/run/udev/links/disk\x2fby-path\x2fpci-0000:00:13.0-ata-1-part2'
sda2: Preserve already existing symlink '/dev/disk/by-path/pci-0000:00:13.0-ata-1-part2' to '../../sda2'
sda2: Found 'b8:2' claiming '/run/udev/links/disk\x2fby-label\x2froot'
sda2: Preserve already existing symlink '/dev/disk/by-label/root' to '../../sda2'
sda2: Found 'b8:2' claiming '/run/udev/links/disk\x2fby-uuid\x2fb0828b38-a6c0-4d3f-abb9-b6e98bc378be'
sda2: Preserve already existing symlink '/dev/disk/by-uuid/b0828b38-a6c0-4d3f-abb9-b6e98bc378be' to '../../sda2'
sda2: Found 'b8:2' claiming '/run/udev/links/disk\x2fby-id\x2fata-Samsung_SSD_860_EVO_500GB_S3Z1NB0K102757R-part2'
sda2: Preserve already existing symlink '/dev/disk/by-id/ata-Samsung_SSD_860_EVO_500GB_S3Z1NB0K102757R-part2' to '../../sda2'
sda2: Preserve already existing symlink '/dev/block/8:2' to '../sda2'
DEVPATH=/devices/pci0000:00/0000:00:13.0/ata1/host0/target0:0:0/0:0:0:0/block/sda/sda2
DEVNAME=/dev/sda2
DEVTYPE=partition
PARTN=2
PARTNAME=/
MAJOR=8
MINOR=2
ACTION=add
SUBSYSTEM=block
TAGS=:systemd:
ID_ATA=1
ID_TYPE=disk
ID_BUS=ata
ID_MODEL=Samsung_SSD_860_EVO_500GB
ID_MODEL_ENC=Samsung\x20SSD\x20860\x20EVO\x20500GB\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20
ID_REVISION=RVT04B6Q
ID_SERIAL=Samsung_SSD_860_EVO_500GB_S3Z1NB0K102757R
ID_SERIAL_SHORT=S3Z1NB0K102757R
ID_ATA_WRITE_CACHE=1
ID_ATA_WRITE_CACHE_ENABLED=1
ID_ATA_FEATURE_SET_HPA=1
ID_ATA_FEATURE_SET_HPA_ENABLED=1
ID_ATA_FEATURE_SET_PM=1
ID_ATA_FEATURE_SET_PM_ENABLED=1
ID_ATA_FEATURE_SET_SECURITY=1
ID_ATA_FEATURE_SET_SECURITY_ENABLED=0
ID_ATA_FEATURE_SET_SECURITY_ERASE_UNIT_MIN=4
ID_ATA_FEATURE_SET_SECURITY_ENHANCED_ERASE_UNIT_MIN=8
ID_ATA_FEATURE_SET_SECURITY_FROZEN=1
ID_ATA_FEATURE_SET_SMART=1
ID_ATA_FEATURE_SET_SMART_ENABLED=1
ID_ATA_DOWNLOAD_MICROCODE=1
ID_ATA_SATA=1
ID_ATA_SATA_SIGNAL_RATE_GEN2=1
ID_ATA_SATA_SIGNAL_RATE_GEN1=1
ID_ATA_ROTATION_RATE_RPM=0
ID_WWN=0x5002538e40060f08
ID_WWN_WITH_EXTENSION=0x5002538e40060f08
ID_PATH=pci-0000:00:13.0-ata-1.0
ID_PATH_TAG=pci-0000_00_13_0-ata-1_0
ID_PATH_ATA_COMPAT=pci-0000:00:13.0-ata-1
ID_PART_TABLE_UUID=c1b68c15-be06-4d7e-bd98-a315f81fd1bf
ID_PART_TABLE_TYPE=gpt
DEVLINKS=/dev/disk/by-partuuid/42632089-53bc-48f5-bded-3c98ea77af8f /dev/disk/by-path/pci-0000:00:13.0-ata-1.0-part2 /dev/disk/by-id/wwn-0x5002538e40060f08-part2 /dev/disk/by-partlabel/\x2f /dev/disk/by-path/pci-0000:00:13.0-ata-1-part2 /dev/disk/by-label/root /dev/disk/by-uuid/b0828b38-a6c0-4d3f-abb9-b6e98bc378be /dev/disk/by-id/ata-Samsung_SSD_860_EVO_500GB_S3Z1NB0K102757R-part2
ID_FS_LABEL=root
ID_FS_LABEL_ENC=root
ID_FS_UUID=b0828b38-a6c0-4d3f-abb9-b6e98bc378be
ID_FS_UUID_ENC=b0828b38-a6c0-4d3f-abb9-b6e98bc378be
ID_FS_VERSION=1.0
ID_FS_TYPE=ext4
ID_FS_USAGE=filesystem
ID_PART_ENTRY_SCHEME=gpt
ID_PART_ENTRY_NAME=\x2f
ID_PART_ENTRY_UUID=42632089-53bc-48f5-bded-3c98ea77af8f
ID_PART_ENTRY_TYPE=4f68bce3-e8cd-4db1-96e7-fbcaf984b709
ID_PART_ENTRY_NUMBER=2
ID_PART_ENTRY_OFFSET=307200
ID_PART_ENTRY_SIZE=976463872
ID_PART_ENTRY_DISK=8:0
CURRENT_TAGS=:systemd:
USEC_INITIALIZED=3199243
Unload module index
Unloaded link configuration context.
nanon@hkg~ $
cphuntington97 commented 2 years ago

Uh what is in /boot (both before an after mounting the EFI partition)? This installation seems a little odd.

😅 /boot is empty. I checked it by booting to a flash drive and while booted into the system.

But the sda1 partition (if that's what you mean?) has:

nanon@hkg~/sda1 $ ls -al
total 15798
drwxr-xr-x 2 root  root      1024 Dec 31  1969 .
drwxr-x--- 6 nanon nanon     4096 May 25 21:17 ..
-rwxr-xr-x 1 root  root   4800512 May 19 10:04 freestanding-00-intel-ucode.cpio
-rwxr-xr-x 1 root  root     33016 May 19 10:04 freestanding-i915-firmware.cpio.xz
-r-xr-xr-x 1 root  root    119328 May 19 10:04 ldlinux.c32
-r-xr-xr-x 1 root  root     61952 May 19 10:04 ldlinux.sys
-rwxr-xr-x 1 root  root  11154624 May 19 10:04 org.clearlinux.native.5.13.13-1070
-rwxr-xr-x 1 root  root       530 May 19 10:04 syslinux.cfg
nanon@hkg~/sda1 $
bryteise commented 2 years ago

Okay, you aren't using EFI which is why the udev output is the way it is (since they depend on an ESP). In that case this is a bug that my initial workaround didn't account for. If you want to resolve this for yourself while I figure out an automated resolution try creating a /etc/systemd/system/-.mount file with the following content (replacing the What with the correct information).

[Unit]
Description=Mount root

[Mount]
What=/dev/sda3
Where=/
Type=ext4
Options=defaults

[Install]
WantedBy=local-fs.target

Then systemctl enable it, reboot and I expect that mount no longer says /dev/root / but instead whatever you have in the What and that clr-boot-manager will work again.

cphuntington97 commented 2 years ago

Thanks so much for looking into this! Alas, I don't think that had the desired effect. This is after 

nnn@enf~ $ sudo systemctl enable /etc/systemd/system/-.mount 
Created symlink /etc/systemd/system/local-fs.target.wants/-.mount → /etc/systemd/system/-.mount.
nnn@enf~ $

and sudo reboot.

nnn@enf~ $ cat /etc/systemd/system/-.mount 
[Unit] 
Description=Mount root 

[Mount] 
What=/dev/vda2 
Where=/ 
Type=ext4 
Options=defaults 

[Install] 
WantedBy=local-fs.target

nnn@enf~ $ lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
vda    254:0    0   25G  0 disk 
├─vda1 254:1    0  149M  0 part 
└─vda2 254:2    0 24.9G  0 part /
nnn@enf~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        25G  2.8G   21G  12% /
devtmpfs        481M     0  481M   0% /dev
tmpfs           482M     0  482M   0% /dev/shm
tmpfs           193M  628K  193M   1% /run
tmpfs           4.0M     0  4.0M   0% /sys/fs/cgroup
tmpfs           482M     0  482M   0% /tmp
nnn@enf~ $ sudo clr-boot-manager list-kernels
Segmentation fault
nnn@enf~ $
bryteise commented 2 years ago

Well darn, not sure why mount is still showing /dev/root and not /dev/vda2.

cphuntington97 commented 2 years ago

@bryteise you've been very gracious. It's up to you if you want to keep troubleshooting this. I can always install as EFI on local hardware; the primary reason to support legacy BIOS would be to support running Clear Linux on VPS hosts like Linode, UpCloud, and Vultr.

Looks like AWS, Azure, and Google Cloud all support UEFI, so I can always take my business there. I have no idea what sort of installed base exists using legacy BIOS, but I imagine it trends toward the modest side.

bryteise commented 2 years ago

Well as much as I'd like to avoid touching legacy BIOS anymore I'm pretty sure that my current workaround could use to be expanded for potentially other cases where the initrd causes /dev/root to set as the rootfs device. Hopefully will have something later in the week.

cphuntington97 commented 2 years ago

Hopefully will have something later in the week.

I'm looking forward to it!

bryteise commented 2 years ago

Looks like the change won't be in until next week's build unfortunately but otherwise as far as I've tested it is working.

cphuntington97 commented 2 years ago

Looks like the change won't be in until next week's build unfortunately but otherwise as far as I've tested it is working.

I just want to confirm that it is fixed on all systems here. Thank you!