search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
18.29k stars 14.27k forks source link

bCacheFS as root that worked (and continues to work) on 23.11 fails on 24.05 #316396

Open Krutonium opened 5 months ago

Krutonium commented 5 months ago

Describe the bug

https://gitea.krutonium.ca/Krutonium/NixOS/src/commit/8943c3814582f48318142712da56384057d6dec0/devices/uMsiLaptop-hw.nix#L17-L26

On 23.11 this worked just fine, the system (admittedly slowly) mounted the bCacheFS root filesystem and successfully booted. On 24.05, the system fails to mount the filesystem and then kernel panics. The filesystem is a 2 disk filesystem, one ssd and one hdd, with compression enabled.

Steps To Reproduce

Steps to reproduce the behavior:

  1. Create a Multi Disk bCacheFS and use it as root partition on 23.11, Mounted by UUID
  2. Upgrade to 24.05

Expected behavior

The system should boot as normal.

Screenshots

20240601_072214 20240601_072038 20240601_072118 20240601_072144

Additional context

I am using a 2 disk filesystem with the goal of having lots of fast storage. It's working fine, albeit with high memory usage, but after the upgrade it seems to fail to boot.

Notify maintainers

I don't honestly know who to notify here.

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

✔ >>  nix-shell -p nix-info --run "nix-info -m"

 - system: `"x86_64-linux"`
 - host os: `Linux 6.9.2, NixOS, 23.11 (Tapir), 23.11.20240525.9d29cd2`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.1`
 - nixpkgs: `/nix/store/f6lbic2a83c51ygb2czksw9gv8x6w5wg-source`

Add a :+1: reaction to issues you find important.

Krutonium commented 5 months ago

Only when you take pictures with the intention of sharing do you realize how utterly filthy your display is. Rest assured that this will be rectified.

JohnRTitor commented 5 months ago

Hi there, could you try running fsck from a nix iso using the latest bcachefs-tools?

nix run github:koverstreet/bcachefs-tools#bcachefs-tools -- help

Then boot in your generation with 24.05.

Krutonium commented 5 months ago

Will do!

Krutonium commented 5 months ago

Okay so after running the fsck, the system now boots, but it still has all those crashes during bootup. (The same ones in fact)

JohnRTitor commented 5 months ago

Which specific ones? Please post new screenshots

Krutonium commented 5 months ago

boot.log I'll do you one better (relevant bit copied from bootlog)

Jun 03 04:55:19 uMsiLaptop kernel: bcachefs (UUID=1f35e804-826b-4fe8-926e-7545b08c383f): error reading superblock: error opening UUID=1f35e804-826b-4fe8-926e-7545b08c383f: ENOENT
Jun 03 04:55:19 uMsiLaptop kernel: bcachefs (1f35e804-826b-4fe8-926e-7545b08c383f): mounting version 1.7: mi_btree_bitmap opts=compression=lz4,background_compression=zstd:15,foreground_target=ssd,background_target=hdd,promote_target=ssd
Jun 03 04:55:19 uMsiLaptop kernel: bcachefs (1f35e804-826b-4fe8-926e-7545b08c383f): recovering from clean shutdown, journal seq 23350044
Jun 03 04:55:19 uMsiLaptop kernel: bcachefs (1f35e804-826b-4fe8-926e-7545b08c383f): alloc_read... done
Jun 03 04:55:19 uMsiLaptop kernel: bcachefs (1f35e804-826b-4fe8-926e-7545b08c383f): stripes_read... done
Jun 03 04:55:19 uMsiLaptop kernel: bcachefs (1f35e804-826b-4fe8-926e-7545b08c383f): snapshots_read... done
Jun 03 04:55:19 uMsiLaptop kernel: bcachefs (1f35e804-826b-4fe8-926e-7545b08c383f): journal_replay... done
Jun 03 04:55:19 uMsiLaptop kernel: bcachefs (1f35e804-826b-4fe8-926e-7545b08c383f): resume_logged_ops... done
Jun 03 04:55:19 uMsiLaptop kernel: bcachefs (1f35e804-826b-4fe8-926e-7545b08c383f): going read-write
Jun 03 04:55:19 uMsiLaptop unknown: booting system configuration /nix/store/p1af182snji67ay61dr8hvsnnpcp65ri-nixos-system-uMsiLaptop-24.05.20240531.805a384
Jun 03 04:55:19 uMsiLaptop stage-2-init: running activation script...
Jun 03 04:55:19 uMsiLaptop kernel: ------------[ cut here ]------------
Jun 03 04:55:19 uMsiLaptop kernel: btree trans held srcu lock (delaying memory reclaim) for 17 seconds
Jun 03 04:55:19 uMsiLaptop kernel: WARNING: CPU: 1 PID: 472 at fs/bcachefs/btree_iter.c:2871 bch2_trans_put+0x23e/0x270 [bcachefs]
Jun 03 04:55:19 uMsiLaptop kernel: Modules linked in: bcachefs libcrc32c crc32c_generic lz4_compress lz4hc_compress xor raid6_pq sd_mod sr_mod t10_pi cdrom crc64_rocksoft crc_t10dif crct10dif_generic crc64 ahci libahci rtsx_pci_sdmmc libata mmc_core atkbd xhci_pci libps2 xhci_pci_renesas scsi_mod vivaldi_fmap xhci_hcd rtsx_pci ehci_pci crct10dif_pclmul crct10dif_common crc32c_intel ehci_hcd sha256_ssse3 scsi_common i8042 rtc_cmos serio dm_mod dax i915 i2c_algo_bit drm_buddy video wmi backlight ttm intel_gtt drm_display_helper firmware_class cec
Jun 03 04:55:19 uMsiLaptop kernel: CPU: 1 PID: 472 Comm: ln Not tainted 6.9.2 #1-NixOS
Jun 03 04:55:19 uMsiLaptop kernel: Hardware name: Micro-Star International Co., Ltd. GP60 2QF/MS-16GH, BIOS E16GHIMS.10I 03/12/2015
Jun 03 04:55:19 uMsiLaptop kernel: RIP: 0010:bch2_trans_put+0x23e/0x270 [bcachefs]
Jun 03 04:55:19 uMsiLaptop kernel: Code: 96 ce 48 c7 c7 78 c6 df c0 48 b8 cf f7 53 e3 a5 9b c4 20 48 29 ca 48 c1 ea 03 48 f7 e2 48 89 d6 48 c1 ee 04 e8 13 ff e0 cc 90 <0f> 0b 90 90 8b b5 a8 00 00 00 49 8d be 68 36 00 00 83 fe 01 77 0a
Jun 03 04:55:19 uMsiLaptop kernel: RSP: 0018:ffff96f202257bb0 EFLAGS: 00010286
Jun 03 04:55:19 uMsiLaptop kernel: RAX: 0000000000000000 RBX: ffff8dd1804d5880 RCX: 0000000000000027
Jun 03 04:55:19 uMsiLaptop kernel: RDX: ffff8dd49f8a1848 RSI: 0000000000000001 RDI: ffff8dd49f8a1840
Jun 03 04:55:19 uMsiLaptop kernel: RBP: ffff8dd18c3dc000 R08: 0000000000000000 R09: 0000000000000003
Jun 03 04:55:19 uMsiLaptop kernel: R10: ffff96f202257a58 R11: ffffffff8f73a128 R12: ffff8dd18c3dc000
Jun 03 04:55:19 uMsiLaptop kernel: R13: ffff96f202257cf0 R14: ffff8dd18d300000 R15: ffff8dd1804d5880
Jun 03 04:55:19 uMsiLaptop kernel: FS:  00007fda88d43140(0000) GS:ffff8dd49f880000(0000) knlGS:0000000000000000
Jun 03 04:55:19 uMsiLaptop kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 03 04:55:19 uMsiLaptop kernel: CR2: 00007f2a705e06d0 CR3: 00000001057f4006 CR4: 00000000001706f0
Jun 03 04:55:19 uMsiLaptop kernel: Call Trace:
Jun 03 04:55:19 uMsiLaptop kernel:  <TASK>
Jun 03 04:55:19 uMsiLaptop kernel:  ? __warn+0x80/0x120
Jun 03 04:55:19 uMsiLaptop kernel:  ? bch2_trans_put+0x23e/0x270 [bcachefs]
Jun 03 04:55:19 uMsiLaptop kernel:  ? report_bug+0x164/0x190
Jun 03 04:55:19 uMsiLaptop kernel:  ? handle_bug+0x3d/0x80
Jun 03 04:55:19 uMsiLaptop kernel:  ? exc_invalid_op+0x17/0x70
Jun 03 04:55:19 uMsiLaptop kernel:  ? asm_exc_invalid_op+0x1a/0x20
Jun 03 04:55:19 uMsiLaptop kernel:  ? bch2_trans_put+0x23e/0x270 [bcachefs]
Jun 03 04:55:19 uMsiLaptop kernel:  __bch2_create+0x4d5/0x5c0 [bcachefs]
Jun 03 04:55:19 uMsiLaptop kernel:  ? bch2_symlink+0x49/0x110 [bcachefs]
Jun 03 04:55:19 uMsiLaptop kernel:  bch2_symlink+0x49/0x110 [bcachefs]
Jun 03 04:55:19 uMsiLaptop kernel:  vfs_symlink+0x13b/0x1d0
Jun 03 04:55:19 uMsiLaptop kernel:  do_symlinkat+0x114/0x140
Jun 03 04:55:19 uMsiLaptop kernel:  __x64_sys_symlinkat+0x3d/0x50
Jun 03 04:55:19 uMsiLaptop kernel:  do_syscall_64+0xb8/0x200
Jun 03 04:55:19 uMsiLaptop kernel:  entry_SYSCALL_64_after_hwframe+0x77/0x7f
Jun 03 04:55:19 uMsiLaptop kernel: RIP: 0033:0x7fda88e4113b
Jun 03 04:55:19 uMsiLaptop kernel: Code: 77 05 c3 0f 1f 40 00 48 8b 15 d9 0c 0e 00 f7 d8 64 89 02 b8 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 0a 01 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d ad 0c 0e 00 f7 d8 64 89 01 48
Jun 03 04:55:19 uMsiLaptop kernel: RSP: 002b:00007ffd58ea8368 EFLAGS: 00000246 ORIG_RAX: 000000000000010a
Jun 03 04:55:19 uMsiLaptop kernel: RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007fda88e4113b
Jun 03 04:55:19 uMsiLaptop kernel: RDX: 00007ffd58ea9b30 RSI: 00000000ffffff9c RDI: 00007ffd58ea9ae5
Jun 03 04:55:19 uMsiLaptop kernel: RBP: 00007ffd58ea8628 R08: 00000000ffffffff R09: 0000000000000000
Jun 03 04:55:19 uMsiLaptop kernel: R10: 00007ffd58ea8618 R11: 0000000000000246 R12: 0000000000517ab4
Jun 03 04:55:19 uMsiLaptop kernel: R13: 00000000004e3d01 R14: 00000000005388e0 R15: 0000000000000000
Jun 03 04:55:19 uMsiLaptop kernel:  </TASK>
Jun 03 04:55:19 uMsiLaptop kernel: ---[ end trace 0000000000000000 ]---
Jun 03 04:55:19 uMsiLaptop stage-2-init: reviving group 'gnome-remote-desktop' with GID 989
Jun 03 04:55:19 uMsiLaptop stage-2-init: removing group ‘systemd-journal-gateway’
Jun 03 04:55:19 uMsiLaptop stage-2-init: removing user ‘systemd-journal-gateway’
Jun 03 04:55:19 uMsiLaptop kernel: ------------[ cut here ]------------
Jun 03 04:55:19 uMsiLaptop kernel: btree trans held srcu lock (delaying memory reclaim) for 17 seconds
Jun 03 04:55:19 uMsiLaptop kernel: WARNING: CPU: 1 PID: 476 at fs/bcachefs/btree_iter.c:2871 bch2_trans_put+0x23e/0x270 [bcachefs]
Jun 03 04:55:19 uMsiLaptop kernel: Modules linked in: bcachefs libcrc32c crc32c_generic lz4_compress lz4hc_compress xor raid6_pq sd_mod sr_mod t10_pi cdrom crc64_rocksoft crc_t10dif crct10dif_generic crc64 ahci libahci rtsx_pci_sdmmc libata mmc_core atkbd xhci_pci libps2 xhci_pci_renesas scsi_mod vivaldi_fmap xhci_hcd rtsx_pci ehci_pci crct10dif_pclmul crct10dif_common crc32c_intel ehci_hcd sha256_ssse3 scsi_common i8042 rtc_cmos serio dm_mod dax i915 i2c_algo_bit drm_buddy video wmi backlight ttm intel_gtt drm_display_helper firmware_class cec
Jun 03 04:55:19 uMsiLaptop kernel: CPU: 1 PID: 476 Comm: perl Tainted: G        W          6.9.2 #1-NixOS
Jun 03 04:55:19 uMsiLaptop kernel: Hardware name: Micro-Star International Co., Ltd. GP60 2QF/MS-16GH, BIOS E16GHIMS.10I 03/12/2015
Jun 03 04:55:19 uMsiLaptop kernel: RIP: 0010:bch2_trans_put+0x23e/0x270 [bcachefs]
Jun 03 04:55:19 uMsiLaptop kernel: Code: 96 ce 48 c7 c7 78 c6 df c0 48 b8 cf f7 53 e3 a5 9b c4 20 48 29 ca 48 c1 ea 03 48 f7 e2 48 89 d6 48 c1 ee 04 e8 13 ff e0 cc 90 <0f> 0b 90 90 8b b5 a8 00 00 00 49 8d be 68 36 00 00 83 fe 01 77 0a
Jun 03 04:55:19 uMsiLaptop kernel: RSP: 0018:ffff96f202677a40 EFLAGS: 00010282
Jun 03 04:55:19 uMsiLaptop kernel: RAX: 0000000000000000 RBX: ffff8dd189db6390 RCX: 0000000000000027
Jun 03 04:55:19 uMsiLaptop kernel: RDX: ffff8dd49f8a1848 RSI: 0000000000000001 RDI: ffff8dd49f8a1840
Jun 03 04:55:19 uMsiLaptop kernel: RBP: ffff8dd18c3e0000 R08: 0000000000000000 R09: 0000000000000003
Jun 03 04:55:19 uMsiLaptop kernel: R10: ffff96f2026778e8 R11: ffffffff8f73a128 R12: ffff8dd18c3e0000
Jun 03 04:55:19 uMsiLaptop kernel: R13: ffff96f202677b80 R14: ffff8dd18d300000 R15: ffff8dd189db6390
Jun 03 04:55:19 uMsiLaptop kernel: FS:  00007f69d217b740(0000) GS:ffff8dd49f880000(0000) knlGS:0000000000000000
Jun 03 04:55:19 uMsiLaptop kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 03 04:55:19 uMsiLaptop kernel: CR2: 00000000339d87fc CR3: 000000010d0ce003 CR4: 00000000001706f0
Jun 03 04:55:19 uMsiLaptop kernel: Call Trace:
Jun 03 04:55:19 uMsiLaptop kernel:  <TASK>
Jun 03 04:55:19 uMsiLaptop kernel:  ? __warn+0x80/0x120
Jun 03 04:55:19 uMsiLaptop kernel:  ? bch2_trans_put+0x23e/0x270 [bcachefs]
Jun 03 04:55:19 uMsiLaptop kernel:  ? report_bug+0x164/0x190
Jun 03 04:55:19 uMsiLaptop kernel:  ? handle_bug+0x3d/0x80
Jun 03 04:55:19 uMsiLaptop kernel:  ? exc_invalid_op+0x17/0x70
Jun 03 04:55:19 uMsiLaptop kernel:  ? asm_exc_invalid_op+0x1a/0x20
Jun 03 04:55:19 uMsiLaptop kernel:  ? bch2_trans_put+0x23e/0x270 [bcachefs]
Jun 03 04:55:19 uMsiLaptop kernel:  __bch2_create+0x4d5/0x5c0 [bcachefs]
Jun 03 04:55:19 uMsiLaptop kernel:  ? bch2_create+0x2a/0x60 [bcachefs]
Jun 03 04:55:19 uMsiLaptop kernel:  bch2_create+0x2a/0x60 [bcachefs]
Jun 03 04:55:19 uMsiLaptop kernel:  path_openat+0xe8a/0x1150
Jun 03 04:55:19 uMsiLaptop kernel:  do_filp_open+0xc4/0x170
Jun 03 04:55:19 uMsiLaptop kernel:  do_sys_openat2+0xab/0xe0
Jun 03 04:55:19 uMsiLaptop kernel:  __x64_sys_openat+0x57/0xa0
Jun 03 04:55:19 uMsiLaptop kernel:  do_syscall_64+0xb8/0x200
Jun 03 04:55:19 uMsiLaptop kernel:  entry_SYSCALL_64_after_hwframe+0x77/0x7f
Jun 03 04:55:19 uMsiLaptop kernel: RIP: 0033:0x7f69d227a2b2
Jun 03 04:55:19 uMsiLaptop kernel: Code: 83 e2 40 75 53 89 f0 f7 d0 a9 00 00 41 00 74 48 80 3d a1 9d 0e 00 00 74 6c 89 da 48 89 ee bf 9c ff ff ff b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 92 00 00 00 48 8b 54 24 28 64 48 2b 14 25
Jun 03 04:55:19 uMsiLaptop kernel: RSP: 002b:00007ffd29fb5e70 EFLAGS: 00000202 ORIG_RAX: 0000000000000101
Jun 03 04:55:19 uMsiLaptop kernel: RAX: ffffffffffffffda RBX: 0000000000080041 RCX: 00007f69d227a2b2
Jun 03 04:55:19 uMsiLaptop kernel: RDX: 0000000000080041 RSI: 0000000033a2f720 RDI: 00000000ffffff9c
Jun 03 04:55:19 uMsiLaptop kernel: RBP: 0000000033a2f720 R08: 00007ffd29fb60f0 R09: 00000000ffffffff
Jun 03 04:55:19 uMsiLaptop kernel: R10: 00000000000001a4 R11: 0000000000000202 R12: 0000000000000000
Jun 03 04:55:19 uMsiLaptop kernel: R13: 0000000033a2f720 R14: 0000000033285ec0 R15: 0000000000000000
Jun 03 04:55:19 uMsiLaptop kernel:  </TASK>
Jun 03 04:55:19 uMsiLaptop kernel: ---[ end trace 0000000000000000 ]---
Jun 03 04:55:19 uMsiLaptop stage-2-init: setting up /etc...
Jun 03 04:55:19 uMsiLaptop kernel: ------------[ cut here ]------------
Jun 03 04:55:19 uMsiLaptop kernel: btree trans held srcu lock (delaying memory reclaim) for 11 seconds
Jun 03 04:55:19 uMsiLaptop kernel: WARNING: CPU: 3 PID: 479 at fs/bcachefs/btree_iter.c:2871 bch2_trans_put+0x23e/0x270 [bcachefs]
Jun 03 04:55:19 uMsiLaptop kernel: Modules linked in: bcachefs libcrc32c crc32c_generic lz4_compress lz4hc_compress xor raid6_pq sd_mod sr_mod t10_pi cdrom crc64_rocksoft crc_t10dif crct10dif_generic crc64 ahci libahci rtsx_pci_sdmmc libata mmc_core atkbd xhci_pci libps2 xhci_pci_renesas scsi_mod vivaldi_fmap xhci_hcd rtsx_pci ehci_pci crct10dif_pclmul crct10dif_common crc32c_intel ehci_hcd sha256_ssse3 scsi_common i8042 rtc_cmos serio dm_mod dax i915 i2c_algo_bit drm_buddy video wmi backlight ttm intel_gtt drm_display_helper firmware_class cec
Jun 03 04:55:19 uMsiLaptop kernel: CPU: 3 PID: 479 Comm: perl Tainted: G        W          6.9.2 #1-NixOS
Jun 03 04:55:19 uMsiLaptop kernel: Hardware name: Micro-Star International Co., Ltd. GP60 2QF/MS-16GH, BIOS E16GHIMS.10I 03/12/2015
Jun 03 04:55:19 uMsiLaptop kernel: RIP: 0010:bch2_trans_put+0x23e/0x270 [bcachefs]
Jun 03 04:55:19 uMsiLaptop kernel: Code: 96 ce 48 c7 c7 78 c6 df c0 48 b8 cf f7 53 e3 a5 9b c4 20 48 29 ca 48 c1 ea 03 48 f7 e2 48 89 d6 48 c1 ee 04 e8 13 ff e0 cc 90 <0f> 0b 90 90 8b b5 a8 00 00 00 49 8d be 68 36 00 00 83 fe 01 77 0a
Jun 03 04:55:19 uMsiLaptop kernel: RSP: 0018:ffff96f2006dbbb0 EFLAGS: 00010286
Jun 03 04:55:19 uMsiLaptop kernel: RAX: 0000000000000000 RBX: ffff8dd182a0f250 RCX: 0000000000000027
Jun 03 04:55:19 uMsiLaptop kernel: RDX: ffff8dd49f9a1848 RSI: 0000000000000001 RDI: ffff8dd49f9a1840
Jun 03 04:55:19 uMsiLaptop kernel: RBP: ffff8dd2645b8000 R08: 0000000000000000 R09: 0000000000000003
Jun 03 04:55:19 uMsiLaptop kernel: R10: ffff96f2006dba58 R11: ffffffff8f73a128 R12: ffff8dd2645b8000
Jun 03 04:55:19 uMsiLaptop kernel: R13: ffff96f2006dbcf0 R14: ffff8dd18d300000 R15: ffff8dd182a0f250
Jun 03 04:55:19 uMsiLaptop kernel: FS:  00007f156941b740(0000) GS:ffff8dd49f980000(0000) knlGS:0000000000000000
Jun 03 04:55:19 uMsiLaptop kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 03 04:55:19 uMsiLaptop kernel: CR2: 000000000cd68c58 CR3: 000000010d0e0001 CR4: 00000000001706f0
Jun 03 04:55:19 uMsiLaptop kernel: Call Trace:
Jun 03 04:55:19 uMsiLaptop kernel:  <TASK>
Jun 03 04:55:19 uMsiLaptop kernel:  ? __warn+0x80/0x120
Jun 03 04:55:19 uMsiLaptop kernel:  ? bch2_trans_put+0x23e/0x270 [bcachefs]
Jun 03 04:55:19 uMsiLaptop kernel:  ? report_bug+0x164/0x190
Jun 03 04:55:19 uMsiLaptop kernel:  ? handle_bug+0x3d/0x80
Jun 03 04:55:19 uMsiLaptop kernel:  ? exc_invalid_op+0x17/0x70
Jun 03 04:55:19 uMsiLaptop kernel:  ? asm_exc_invalid_op+0x1a/0x20
Jun 03 04:55:19 uMsiLaptop kernel:  ? bch2_trans_put+0x23e/0x270 [bcachefs]
Jun 03 04:55:19 uMsiLaptop kernel:  __bch2_create+0x4d5/0x5c0 [bcachefs]
Jun 03 04:55:19 uMsiLaptop kernel:  ? bch2_symlink+0x49/0x110 [bcachefs]
Jun 03 04:55:19 uMsiLaptop kernel:  bch2_symlink+0x49/0x110 [bcachefs]
Jun 03 04:55:19 uMsiLaptop kernel:  vfs_symlink+0x13b/0x1d0
Jun 03 04:55:19 uMsiLaptop kernel:  do_symlinkat+0x114/0x140
Jun 03 04:55:19 uMsiLaptop kernel:  __x64_sys_symlink+0x3b/0x50
Jun 03 04:55:19 uMsiLaptop kernel:  do_syscall_64+0xb8/0x200
Jun 03 04:55:19 uMsiLaptop kernel:  entry_SYSCALL_64_after_hwframe+0x77/0x7f
Jun 03 04:55:19 uMsiLaptop kernel: RIP: 0033:0x7f156951b10b
Jun 03 04:55:19 uMsiLaptop kernel: Code: 8b 0d 11 0d 0e 00 f7 d8 64 89 01 b9 ff ff ff ff eb d3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 f3 0f 1e fa b8 58 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 d9 0c 0e 00 f7 d8
Jun 03 04:55:19 uMsiLaptop kernel: RSP: 002b:00007fffc84a6848 EFLAGS: 00000202 ORIG_RAX: 0000000000000058
Jun 03 04:55:19 uMsiLaptop kernel: RAX: ffffffffffffffda RBX: 000000000c82b2a0 RCX: 00007f156951b10b
Jun 03 04:55:19 uMsiLaptop kernel: RDX: 000000000c867070 RSI: 000000000cd64ff0 RDI: 000000000c85c470
Jun 03 04:55:19 uMsiLaptop kernel: RBP: 000000000c830d10 R08: 000000000cd64ff0 R09: 000000000c830d00
Jun 03 04:55:19 uMsiLaptop kernel: R10: 0000000000000100 R11: 0000000000000202 R12: 000000000ccf2c00
Jun 03 04:55:19 uMsiLaptop kernel: R13: 000000000c830d08 R14: 0000000000000133 R15: 000000000c85c470
Jun 03 04:55:19 uMsiLaptop kernel:  </TASK>
Jun 03 04:55:19 uMsiLaptop kernel: ---[ end trace 0000000000000000 ]---
Jun 03 04:55:19 uMsiLaptop stage-2-init: removing obsolete symlink ‘/etc/pulse/client.conf’...
Jun 03 04:55:19 uMsiLaptop kernel: ------------[ cut here ]------------
Jun 03 04:55:19 uMsiLaptop kernel: btree trans held srcu lock (delaying memory reclaim) for 14 seconds
Jun 03 04:55:19 uMsiLaptop kernel: WARNING: CPU: 2 PID: 498 at fs/bcachefs/btree_iter.c:2871 bch2_trans_put+0x23e/0x270 [bcachefs]
Jun 03 04:55:19 uMsiLaptop kernel: Modules linked in: bcachefs libcrc32c crc32c_generic lz4_compress lz4hc_compress xor raid6_pq sd_mod sr_mod t10_pi cdrom crc64_rocksoft crc_t10dif crct10dif_generic crc64 ahci libahci rtsx_pci_sdmmc libata mmc_core atkbd xhci_pci libps2 xhci_pci_renesas scsi_mod vivaldi_fmap xhci_hcd rtsx_pci ehci_pci crct10dif_pclmul crct10dif_common crc32c_intel ehci_hcd sha256_ssse3 scsi_common i8042 rtc_cmos serio dm_mod dax i915 i2c_algo_bit drm_buddy video wmi backlight ttm intel_gtt drm_display_helper firmware_class cec
Jun 03 04:55:19 uMsiLaptop kernel: CPU: 2 PID: 498 Comm: ln Tainted: G        W          6.9.2 #1-NixOS
Jun 03 04:55:19 uMsiLaptop kernel: Hardware name: Micro-Star International Co., Ltd. GP60 2QF/MS-16GH, BIOS E16GHIMS.10I 03/12/2015
Jun 03 04:55:19 uMsiLaptop kernel: RIP: 0010:bch2_trans_put+0x23e/0x270 [bcachefs]
Jun 03 04:55:19 uMsiLaptop kernel: Code: 96 ce 48 c7 c7 78 c6 df c0 48 b8 cf f7 53 e3 a5 9b c4 20 48 29 ca 48 c1 ea 03 48 f7 e2 48 89 d6 48 c1 ee 04 e8 13 ff e0 cc 90 <0f> 0b 90 90 8b b5 a8 00 00 00 49 8d be 68 36 00 00 83 fe 01 77 0a
Jun 03 04:55:19 uMsiLaptop kernel: RSP: 0018:ffff96f208c83bb0 EFLAGS: 00010286
Jun 03 04:55:19 uMsiLaptop kernel: RAX: 0000000000000000 RBX: ffff8dd189e21270 RCX: 0000000000000027
Jun 03 04:55:19 uMsiLaptop kernel: RDX: ffff8dd49f921848 RSI: 0000000000000001 RDI: ffff8dd49f921840
Jun 03 04:55:19 uMsiLaptop kernel: RBP: ffff8dd18835c000 R08: 0000000000000000 R09: 0000000000000003
Jun 03 04:55:19 uMsiLaptop kernel: R10: ffff96f208c83a58 R11: ffffffff8f73a128 R12: ffff8dd18835c000
Jun 03 04:55:19 uMsiLaptop kernel: R13: ffff96f208c83cf0 R14: ffff8dd18d300000 R15: ffff8dd189e21270
Jun 03 04:55:19 uMsiLaptop kernel: FS:  00007ff402d07140(0000) GS:ffff8dd49f900000(0000) knlGS:0000000000000000
Jun 03 04:55:19 uMsiLaptop kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 03 04:55:19 uMsiLaptop kernel: CR2: 00007ff402de5970 CR3: 00000002a6a48006 CR4: 00000000001706f0
Jun 03 04:55:19 uMsiLaptop kernel: Call Trace:
Jun 03 04:55:19 uMsiLaptop kernel:  <TASK>
Jun 03 04:55:19 uMsiLaptop kernel:  ? __warn+0x80/0x120
Jun 03 04:55:19 uMsiLaptop kernel:  ? bch2_trans_put+0x23e/0x270 [bcachefs]
Jun 03 04:55:19 uMsiLaptop kernel:  ? report_bug+0x164/0x190
Jun 03 04:55:19 uMsiLaptop kernel:  ? handle_bug+0x3d/0x80
Jun 03 04:55:19 uMsiLaptop kernel:  ? exc_invalid_op+0x17/0x70
Jun 03 04:55:19 uMsiLaptop kernel:  ? asm_exc_invalid_op+0x1a/0x20
Jun 03 04:55:19 uMsiLaptop kernel:  ? bch2_trans_put+0x23e/0x270 [bcachefs]
Jun 03 04:55:19 uMsiLaptop kernel:  __bch2_create+0x4d5/0x5c0 [bcachefs]
Jun 03 04:55:19 uMsiLaptop kernel:  ? bch2_symlink+0x49/0x110 [bcachefs]
Jun 03 04:55:19 uMsiLaptop kernel:  bch2_symlink+0x49/0x110 [bcachefs]
Jun 03 04:55:19 uMsiLaptop kernel:  vfs_symlink+0x13b/0x1d0
Jun 03 04:55:19 uMsiLaptop kernel:  do_symlinkat+0x114/0x140
Jun 03 04:55:19 uMsiLaptop kernel:  __x64_sys_symlinkat+0x3d/0x50
Jun 03 04:55:19 uMsiLaptop kernel:  do_syscall_64+0xb8/0x200
Jun 03 04:55:19 uMsiLaptop kernel:  entry_SYSCALL_64_after_hwframe+0x77/0x7f
Jun 03 04:55:19 uMsiLaptop kernel: RIP: 0033:0x7ff402e0513b
Jun 03 04:55:19 uMsiLaptop kernel: Code: 77 05 c3 0f 1f 40 00 48 8b 15 d9 0c 0e 00 f7 d8 64 89 02 b8 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 0a 01 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d ad 0c 0e 00 f7 d8 64 89 01 48
Jun 03 04:55:19 uMsiLaptop kernel: RSP: 002b:00007fff4fdf03c8 EFLAGS: 00000246 ORIG_RAX: 000000000000010a
Jun 03 04:55:19 uMsiLaptop kernel: RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007ff402e0513b
Jun 03 04:55:19 uMsiLaptop kernel: RDX: 00007fff4fdf0b2b RSI: 00000000ffffff9c RDI: 00007fff4fdf0ae9
Jun 03 04:55:19 uMsiLaptop kernel: RBP: 00007fff4fdf0688 R08: 00000000ffffffff R09: 0000000000000000
Jun 03 04:55:19 uMsiLaptop kernel: R10: 00007fff4fdf0678 R11: 0000000000000246 R12: 0000000000517ab4
Jun 03 04:55:19 uMsiLaptop kernel: R13: 00000000004e3d01 R14: 00000000005388e0 R15: 0000000000000000
Jun 03 04:55:19 uMsiLaptop kernel:  </TASK>
Jun 03 04:55:19 uMsiLaptop kernel: ---[ end trace 0000000000000000 ]---
JohnRTitor commented 5 months ago

Looks like the disk version update from 1.3 to 1.7 has been completed successfully. That's why it required fsck and took long time to boot the first time.

I would recommend changing the hardware-configuration.nix like this. That should fix the error reading supeblock bug.

In addition, see if you can reproduce the boot errors with the latest bcachefs-tools from the repository itself. Should be pretty easy if you are using a flake based setup.

First add it to your flake inputs.

    bcachefs-tools.url = "github:koverstreet/bcachefs-tools";

Set up an overlay in configuration.nix: 

  nixpkgs.overlays = [
    (final: prev: {
      bcachefs-tools = inputs.bcachefs-tools.packages.${pkgs.system}.bcachefs-tools;
    })
  ];
Krutonium commented 5 months ago

I had to adapt it to work with my flake setup, but the result is the same. Building now.

Krutonium commented 5 months ago

It did not like that.

More details once I can extract them... If I can

Edit: okay so that crash happened before it was able to write a boot log to disk. Gunna have to resort to screen pictures. FWIW It looked like a list of files, and then it told me the root account was locked.

krypt0nn commented 5 months ago

I also can't boot after updating to NixOS 24.05, although my error messages look a little different.

IMG_20240603_072254

UPD: I managed to solve my problem after a lot of tweaks. It appears that the problem was with the device name in my fileSystems."/".device property. Before I was using UUID=..., it was displaying a few warnings during boot but I didn't pay attention on them because I know there's an old open issue about multi-device support in systemd. However NixOS 24.05 somehow broke compatibility with this UUID=... device name. I discovered some old nixpkgs commit which allows to specify multiple devices using columns, so I gave it a try - and it magically worked! Here's the whole fix in my config file: https://github.com/krypt0nn/dotfiles/commit/bdf430e2b0c43987784b433e90eee28afaeb862c. I just replaced device = "UUID=..." by device = "/dev/sda:/dev/sdb:/dev/nvme0n1p3" and that's it. I hope this will help people with the same problem.

Krutonium commented 5 months ago

20240603_020626.jpg

Krutonium commented 5 months ago

Just so you're aware, if the worst does come to pass and my filesystem utterly dies, nothing of value will be lost. I have well tested backups.

JohnRTitor commented 5 months ago

20240603_020626.jpg

Is that systemd initrd? Could you disable that and just change the hardware configuration to use /dev/disk/by-uiid/ just like I did? Systemd initrd is problematic and it does not have full support of bcachefs yet

JohnRTitor commented 5 months ago

I also can't boot after updating to NixOS 24.05, although my error messages look a little different.

IMG_20240603_072254

https://github.com/NixOS/nixpkgs/issues/316396#issuecomment-2143901512

Krutonium commented 5 months ago

Is that systemd initrd? Could you disable that and just change the hardware configuration to use /dev/disk/by-uiid/ just like I did? Systemd initrd is problematic and it does not have full support of bcachefs yet

Yes, I turned it on since the commit you linked did as well. I'll turn it off again and try again. :+1:

Krutonium commented 5 months ago

Same errors so far (from before systemd initrd), though it did complain about insufficient devices to start before mounting.

Krutonium commented 5 months ago

boot.log

JohnRTitor commented 5 months ago

If you are experiencing this bug with the bcachefs-tools master branch overlay, I'd recommend reporting this upstream.

CC: @tmuehlbacher

Krutonium commented 5 months ago

For reference,

https://gitea.krutonium.ca/Krutonium/NixOS/src/commit/4f4a687fd644ffbc203b9e444a5a2cec3c4b2f4e/flake.nix#L71-L73 https://gitea.krutonium.ca/Krutonium/NixOS/src/commit/4f4a687fd644ffbc203b9e444a5a2cec3c4b2f4e/flake.nix#L64 https://gitea.krutonium.ca/Krutonium/NixOS/src/commit/4f4a687fd644ffbc203b9e444a5a2cec3c4b2f4e/devices/uMsiLaptop-hw.nix#L17-L26

Krutonium commented 5 months ago

If you are experiencing this bug with the bcachefs-tools master branch overlay, I'd recommend reporting this upstream.

CC: @tmuehlbacher

Which details would be relevant to include in the upstream issue report?

JohnRTitor commented 5 months ago

I'd assume boot logs, like you included here, fstab, your kernel version. And the output of bcachefs show-super

Krutonium commented 5 months ago

Well, I was experimenting, and I can no longer update my system. Got any tips?

error:
       … while fetching the input 'git+file:///home/krutonium/NixOS'

       error: cannot link '/nix/store/.tmp-link-4476-1889678134' to '/nix/store/.links/12jzb6nkzqxb2rv9gns3qmbzagfz9ycmkz6hrr106rbjk4xr8c74': Invalid cross-device link
┏━ 1 Errors: 
┃ error:
┃        … while fetching the input 'git+file:///home/krutonium/NixOS'
┃ 
┃        error: cannot link '/nix/store/.tmp-link-4476-1889678134' to '/nix/store/.links/12jzb6nkzqxb2rv9gns3qmbzagfz9ycmkz6hrr106rbjk4xr8c74': Invalid cross-device link
┣━━━                                                            
┗━ ∑ ⚠ Exited with 1 errors reported by nix at 03:39:05 after 0s
JohnRTitor commented 5 months ago

Try with:

sudo nix-store --verify --check-contents --repair
Krutonium commented 5 months ago

Try with:

sudo nix-store --verify --check-contents --repair
 sudo nix-store --verify --check-contents --repair
reading the Nix store...
checking path existence...
checking link hashes...
checking store hashes...

No luck :\

tmuehlbacher commented 5 months ago

That doesn't look too good. Not super sure about how to go about fixing this.

The concrete error about not being able to do hardlinks between /nix/store and /nix/store/.links because they're on different fs (I don't assume they actually are, right?) could probably be worked around temporarily by adding this to the nix command --option auto-optimise-store false or changing nix.conf accordingly.

Krutonium commented 5 months ago

That doesn't look too good. Not super sure about how to go about fixing this.

The concrete error about not being able to do hardlinks between /nix/store and /nix/store/.links because they're on different fs (I don't assume they actually are, right?) could probably be worked around temporarily by adding this to the nix command --option auto-optimise-store false or changing nix.conf accordingly.

They are on the same drive, at least from the perspective of they're both on the same partition under the umbrella of bCache.

JohnRTitor commented 5 months ago

Yes, and that is the reason systemd initrd could not mount both disks at once. Hopefully that fixes with the 256 update.

rodrgz commented 5 months ago

My auxiliary HD with encrypted bcachefs fails to boot after the systemd 255.4 to 255.6, I don't know if it is the root cause.

Jun 04 17:09:56 localhost systemd-cryptsetup[261]: Device contains ambiguous signatures, cannot auto-recover LUKS2. Jun 04 17:09:56 localhost systemd-cryptsetup[261]: Please run "cryptsetup repair" for recovery. Jun 04 17:09:56 localhost systemd-cryptsetup[261]: Failed to load LUKS superblock on device /dev/disk/by-uuid/****----****: Invalid argument

tmuehlbacher commented 5 months ago

Oh, do you have bcachefs inside of a LUKS container or do you use the native encryption feature from bcachefs?

Or do you use boot.initrd.luks.devices.... somewhere?

Krutonium commented 5 months ago

Just an update, I need this laptop to be working properly in the near future, so I will be formatting it later today. I hope that what has happened in this thread (and the resulting upstream issue) helps to correct such issues in the future.