Closed g2p closed 1 month ago
Similar report from someone on IRC (69558c638c465a79be3a08bfeb3d5a15979cbe42 which is current bcachefs/master)
Are you using overlayfs as well?
No; it's configured out.
# CONFIG_OVERLAY_FS is not set
Bisection points to 86d81ec5f5f05846c7c6e48ffb964b24cba2e669 introducing the bug.
Here is the bisection log:
# bad: [69558c638c465a79be3a08bfeb3d5a15979cbe42] bcachefs: fix ei_update_lock lock ordering
# good: [e5c368e86712e939b9691527d3c9becfab6c3dd4] bcachefs: Improve "unable to allocate journal write" message
git bisect start '69558c638c465a79be3a08bfeb3d5a15979cbe42' 'e5c368e86712e939b9691527d3c9becfab6c3dd4'
# good: [89d21b69b4f88e7a04b66bec38a01470cd40d703] bcachefs: Add missing bch2_journal_do_writes() call
git bisect good 89d21b69b4f88e7a04b66bec38a01470cd40d703
# bad: [3bdb0737820fc89af25e200ea1cb5310ddd8f533] bcachefs: Convert bch2_replicas_gc2() to new accounting
git bisect bad 3bdb0737820fc89af25e200ea1cb5310ddd8f533
# bad: [965bbbd32d37e18a50c742117d162b29947d1b57] bcachefs: Align the display format of `btrees/inodes/keys`
git bisect bad 965bbbd32d37e18a50c742117d162b29947d1b57
# good: [a2d23f3d916bf9abd77944882cba131af1085bcc] bcachefs: io clock: run timer fns under clock lock
git bisect good a2d23f3d916bf9abd77944882cba131af1085bcc
# bad: [f49d2c9835f95fa078ea8a8eba6de9cbddb9eb33] bcachefs: Warn on attempting a move with no replicas
git bisect bad f49d2c9835f95fa078ea8a8eba6de9cbddb9eb33
# bad: [86d81ec5f5f05846c7c6e48ffb964b24cba2e669] bcachefs: Mark bch_inode_info as SLAB_ACCOUNT
git bisect bad 86d81ec5f5f05846c7c6e48ffb964b24cba2e669
# good: [29f1c1ae6d2fff3bf4f89d265f4a1a7c8ab78a8e] closures: fix closure_sync + closure debugging
git bisect good 29f1c1ae6d2fff3bf4f89d265f4a1a7c8ab78a8e
# good: [b02f973e67589cf617f229250e2a738ab62ca666] bcachefs: Fix bch2_inode_insert() race path for tmpfiles
git bisect good b02f973e67589cf617f229250e2a738ab62ca666
# first bad commit: [86d81ec5f5f05846c7c6e48ffb964b24cba2e669] bcachefs: Mark bch_inode_info as SLAB_ACCOUNT
Here is dmesg+pstore from the earliest crashing commit: https://bpa.st/KLAD6UTVYZ6TXJHZX3XVRW6GSA
Relevant bits:
<1>[ 131.700092] BUG: kernel NULL pointer dereference, address: 0000000000000008
<1>[ 131.700135] #PF: supervisor read access in kernel mode
<1>[ 131.700159] #PF: error_code(0x0000) - not-present page
<6>[ 131.700184] PGD 0 P4D 0
<4>[ 131.700205] Oops: Oops: 0000 [#1] PREEMPT SMP NOPTI
<4>[ 131.700229] CPU: 1 PID: 5620 Comm: umount Tainted: G E 6.10.0-rc4-g2p #20
<4>[ 131.700263] Hardware name: To Be Filled By O.E.M. X570 Phantom Gaming 4/X570 Phantom Gaming 4, BIOS P5.61 02/22/2024
<4>[ 131.700297] RIP: 0010:list_lru_add+0xa1/0x140
<4>[ 131.700320] Code: b6 47 1c 0f 84 ad 00 00 00 48 8b 75 d0 8b 96 a8 06 00 00 3c 01 0f 87 a8 00 00 00 85 d2 78 04 a8 01 75 6d 4d 03 27 49 83 c4 08 <49> 8b 44 24 08 49 89 5c 24 08 4c 89 23 48 89 43 08 48 89 18 49 8b
Panic#2 Part5
<4>[ 131.700370] RSP: 0018:ffff9b1a43bf7b10 EFLAGS: 00010246
<4>[ 131.700390] RAX: 0000000000000000 RBX: ffff8fabb20df098 RCX: 0000000000000000
<4>[ 131.700412] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
<4>[ 131.700433] RBP: ffff9b1a43bf7b50 R08: 0000000000000000 R09: 0000000000000000
<4>[ 131.700455] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
<4>[ 131.700476] R13: 0000000000000000 R14: ffff8fab811b1040 R15: ffff8fab8c3cacb0
<4>[ 131.700499] FS: 00007fa053478800(0000) GS:ffff8faeaec80000(0000) knlGS:0000000000000000
<4>[ 131.700521] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<4>[ 131.700541] CR2: 0000000000000008 CR3: 0000000105cbe000 CR4: 0000000000350ef0
<4>[ 131.700565] Call Trace:
<4>[ 131.700580] <TASK>
<4>[ 131.700597] ? show_regs+0x70/0x90
<4>[ 131.700616] ? __die+0x28/0x80
<4>[ 131.700632] ? page_fault_oops+0x290/0x5c0
<4>[ 131.700652] ? srso_return_thunk+0x5/0x5f
<4>[ 131.700677] ? do_user_addr_fault+0x308/0x6e0
<4>[ 131.700698] ? srso_return_thunk+0x5/0x5f
<4>[ 131.700718] ? exc_page_fault+0x81/0x1a0
<4>[ 131.700738] ? asm_exc_page_fault+0x2b/0x30
<4>[ 131.700764] ? list_lru_add+0xa1/0x140
<4>[ 131.700786] list_lru_add_obj+0x71/0xa0
<4>[ 131.700806] iput+0x22a/0x260
<4>[ 131.700822] dentry_unlink_inode+0xd4/0x150
<4>[ 131.700842] __dentry_kill+0x73/0x180
<4>[ 131.700861] dput+0xf3/0x1b0
<4>[ 131.700876] shrink_dcache_for_umount+0x85/0x140
<4>[ 131.700897] generic_shutdown_super+0x25/0x180
Panic#2 Part4
<4>[ 131.700918] bch2_kill_sb+0x1a/0x30
<4>[ 131.700938] deactivate_locked_super+0x39/0xc0
<4>[ 131.700958] deactivate_super+0x4a/0x60
<4>[ 131.700977] cleanup_mnt+0xc3/0x170
<4>[ 131.700996] __cleanup_mnt+0x16/0x20
<4>[ 131.701012] task_work_run+0x64/0xa0
<4>[ 131.701030] syscall_exit_to_user_mode+0x171/0x180
<4>[ 131.701052] do_syscall_64+0x74/0x120
<4>[ 131.701071] ? srso_return_thunk+0x5/0x5f
<4>[ 131.701089] ? mntput_no_expire+0x51/0x260
<4>[ 131.701107] ? generic_permission+0x3d/0x230
<4>[ 131.701619] ? srso_return_thunk+0x5/0x5f
<4>[ 131.701992] ? mntput+0x28/0x50
<4>[ 131.702361] ? srso_return_thunk+0x5/0x5f
<4>[ 131.702723] ? path_put+0x22/0x30
<4>[ 131.703083] ? srso_return_thunk+0x5/0x5f
<4>[ 131.703438] ? do_faccessat+0x1b8/0x2f0
<4>[ 131.703795] ? srso_return_thunk+0x5/0x5f
<4>[ 131.704146] ? syscall_exit_to_user_mode+0x7c/0x180
<4>[ 131.704498] ? srso_return_thunk+0x5/0x5f
<4>[ 131.704839] ? do_syscall_64+0x74/0x120
<4>[ 131.705173] ? srso_return_thunk+0x5/0x5f
<4>[ 131.705488] ? srso_return_thunk+0x5/0x5f
<4>[ 131.705784] ? syscall_exit_to_user_mode+0x7c/0x180
<4>[ 131.706081] ? srso_return_thunk+0x5/0x5f
<4>[ 131.706373] ? do_syscall_64+0x74/0x120
<4>[ 131.706664] ? srso_return_thunk+0x5/0x5f
<4>[ 131.706954] ? exc_page_fault+0x8d/0x1a0
<4>[ 131.707248] entry_SYSCALL_64_after_hwframe+0x55/0x5d
<4>[ 131.707539] RIP: 0033:0x7fa05332a9fb
<4>[ 131.707839] Code: c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 31 f6 e9 05 00 00 00 0f 1f 44 00 00 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 e9 83 0d 00 f7 d8
Panic#2 Part3
<4>[ 131.708462] RSP: 002b:00007ffe4dc6de38 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6
<4>[ 131.708782] RAX: 0000000000000000 RBX: 0000564169c45940 RCX: 00007fa05332a9fb
<4>[ 131.709106] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000564169c4ba00
<4>[ 131.709430] RBP: 00007ffe4dc6df10 R08: 00007fa053403b20 R09: 0000000000000020
<4>[ 131.709760] R10: 0000000000000001 R11: 0000000000000246 R12: 0000564169c45a40
<4>[ 131.710092] R13: 0000000000000000 R14: 0000564169c4ba00 R15: 0000564169c4ba40
<4>[ 131.710430] </TASK>
<4>[ 131.715829] CR2: 0000000000000008
<4>[ 131.716285] ---[ end trace 0000000000000000 ]---
<4>[ 131.874923] RIP: 0010:list_lru_add+0xa1/0x140
<4>[ 131.875394] Code: b6 47 1c 0f 84 ad 00 00 00 48 8b 75 d0 8b 96 a8 06 00 00 3c 01 0f 87 a8 00 00 00 85 d2 78 04 a8 01 75 6d 4d 03 27 49 83 c4 08 <49> 8b 44 24 08 49 89 5c 24 08 4c 89 23 48 89 43 08 48 89 18 49 8b
<4>[ 131.876376] RSP: 0018:ffff9b1a43bf7b10 EFLAGS: 00010246
<4>[ 131.876874] RAX: 0000000000000000 RBX: ffff8fabb20df098 RCX: 0000000000000000
<4>[ 131.877378] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
<4>[ 131.877882] RBP: ffff9b1a43bf7b50 R08: 0000000000000000 R09: 0000000000000000
<4>[ 131.878386] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
<4>[ 131.878891] R13: 0000000000000000 R14: ffff8fab811b1040 R15: ffff8fab8c3cacb0
<4>[ 131.879399] FS: 00007fa053478800(0000) GS:ffff8faeaec80000(0000) knlGS:0000000000000000
<4>[ 131.879914] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<4>[ 131.880430] CR2: 0000000000000008 CR3: 0000000105cbe000 CR4: 0000000000350ef0
<0>[ 131.880952] Kernel panic - not syncing: Fatal exception
<0>[ 131.881795] Kernel Offset: 0x1600000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
#20
at the start of the log is for: linux-upstream_6.10.0-rc4-00051-g86d81ec5f5f0-20_amd64.changes
LC_ALL=C scripts/faddr2line linux-image-6.10.0-rc4-g2p-dbg_6.10.0-rc4-00051-g86d81ec5f5f0-20_amd64/usr/lib/debug/lib/modules/6.10.0-rc4-g2p/vmlinux list_lru_add+0xa1/0x140
list_lru_add+0xa1/0x140:
list_add_tail at include/linux/list.h:183
(inlined by) list_lru_add at mm/list_lru.c:97
To confirm further, b13d7b4092d9da1b12e3cc4befc32095e8d101b5 ("bcachefs: bch2_btree_reserve_cache_to_text()" on bcachefs/bcachefs-for-upstream) + revert of 86d81ec5f5f05846c7c6e48ffb964b24cba2e669 rebased over linus/master (as of 8a18fda0febb7790de20ec1c3b4522ce026be1c6) shows no issue.
This first boot is with bcachefs-testing as of 088b4ac0dcd13550ee25d90a19a9e19fe3091e95 rebased on v6.10-rc7.
Kernel messages
https://bpa.st/YA44PYGDBHXSEZH3VYVE72XZG4 Relevant bits (/tmp is bcachefs): ``` <30>[ 6971.810299] (sd-remount)[220825]: Remounting '/tmp' read-only with options 'metadata_checksum=xxhash,data_checksum=xxhash,compression=lz4,background_compression=zstd'. <1>[ 6971.810930] BUG: kernel NULL pointer dereference, address: 0000000000000008 Panic#2 Part5 <1>[ 6971.810940] #PF: supervisor read access in kernel mode <1>[ 6971.810945] #PF: error_code(0x0000) - not-present page <6>[ 6971.810950] PGD 0 P4D 0 <4>[ 6971.810956] Oops: Oops: 0000 [#1] PREEMPT SMP NOPTI <4>[ 6971.810962] CPU: 0 PID: 220825 Comm: (sd-remount) Tainted: G E 6.10.0-rc7-g2p #11 <4>[ 6971.810971] Hardware name: To Be Filled By O.E.M. X570 Phantom Gaming 4/X570 Phantom Gaming 4, BIOS P5.61 02/22/2024 <4>[ 6971.810979] RIP: 0010:list_lru_add+0xa1/0x140 <4>[ 6971.810988] Code: b6 47 1c 0f 84 ad 00 00 00 48 8b 75 d0 8b 96 a8 06 00 00 3c 01 0f 87 a8 00 00 00 85 d2 78 04 a8 01 75 6d 4d 03 27 49 83 c4 08 <49> 8b 44 24 08 49 89 5c 24 08 4c 89 23 48 89 43 08 48 89 18 49 8b <4>[ 6971.811000] RSP: 0018:ffff9e6e4717fbe8 EFLAGS: 00010246 <4>[ 6971.811006] RAX: 0000000000000000 RBX: ffff921b43694040 RCX: 0000000000000000 <4>[ 6971.811012] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 <4>[ 6971.811018] RBP: ffff9e6e4717fc28 R08: 0000000000000000 R09: 0000000000000000 <4>[ 6971.811024] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 <4>[ 6971.811029] R13: 0000000000000000 R14: ffff921b40d4edc0 R15: ffff921b7412fcb0 <4>[ 6971.811035] FS: 00007f53c1415440(0000) GS:ffff921e6ec00000(0000) knlGS:0000000000000000 <4>[ 6971.811042] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 <4>[ 6971.811048] CR2: 0000000000000008 CR3: 0000000118754000 CR4: 0000000000350ef0 <4>[ 6971.811054] Call Trace: <4>[ 6971.811058]This next boot is with bcachefs-for-upstream as of 4d5a912b2d943a49e27edc918a6bd23dd00099bf rebased on v6.10-rc7.
It downgraded to bcachefs 1.7 format, fixed some fragmentation_lru issues, then crashed as before.
Kernel messages
https://bpa.st/BWQ3FG4TO4VW4BP5Y7RLTMRBW4 Relevant bits from pstore at the end: ``` <1>[34554.003252] BUG: kernel NULL pointer dereference, address: 0000000000000008 <1>[34554.003272] #PF: supervisor read access in kernel mode <1>[34554.003283] #PF: error_code(0x0000) - not-present page <6>[34554.003294] PGD 0 P4D 0 <4>[34554.003306] Oops: Oops: 0000 [#1] PREEMPT SMP NOPTI <4>[34554.003319] CPU: 8 PID: 118 Comm: kswapd0 Tainted: G W E 6.10.0-rc7-g2p #12 <4>[34554.003335] Hardware name: To Be Filled By O.E.M. X570 Phantom Gaming 4/X570 Phantom Gaming 4, BIOS P5.61 02/22/2024 <4>[34554.003352] RIP: 0010:list_lru_add+0xa1/0x140 <4>[34554.003369] Code: b6 47 1c 0f 84 ad 00 00 00 48 8b 75 d0 8b 96 a8 06 00 00 3c 01 0f 87 a8 00 00 00 85 d2 78 04 a8 01 75 6d 4d 03 27 49 83 c4 08 <49> 8b 44 24 08 49 89 5c 24 08 4c 89 23 48 89 43 08 48 89 18 49 8b <4>[34554.003396] RSP: 0018:ffff9f9a0047f970 EFLAGS: 00010246 <4>[34554.003410] RAX: 0000000000000000 RBX: ffff8ef268cdc7b0 RCX: 0000000000000000 <4>[34554.003423] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 Oops#1 Part3 <4>[34554.003436] RBP: ffff9f9a0047f9b0 R08: 0000000000000000 R09: 0000000000000000 <4>[34554.003448] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 <4>[34554.003461] R13: 0000000000000000 R14: ffff8ef241ea0b40 R15: ffff8ef246cf2cb0 <4>[34554.003474] FS: 0000000000000000(0000) GS:ffff8ef56f000000(0000) knlGS:0000000000000000 <4>[34554.003490] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 <4>[34554.003503] CR2: 0000000000000008 CR3: 0000000106450000 CR4: 0000000000350ef0 <4>[34554.003515] Call Trace: <4>[34554.003523]I have more crashes like these, some of them soon after mounting and others on shutdown.
I have reverted to bcachefs/master as of e5c368e86712e939b9691527d3c9becfab6c3dd4 (plus v6.10-rc5) which doesn't have the issue.