search

koverstreet / bcachefs

Other
662 stars 70 forks source link

can not mount in the latest version with snapshots built in kernel module #575

Closed bhzhu203 closed 10 months ago

bhzhu203 commented 1 year ago

version b677b15b743a5800ee7bd65538e366956c056ad5


[   15.320910] bcachefs: loading out-of-tree module taints kernel.
[   15.404908] bcachefs (vdb): mounting version 1.1: snapshot_skiplists opts=compression=lz4
[   15.404936] bcachefs (vdb): recovering from clean shutdown, journal seq 29271258
[   15.440876] bcachefs (vdb): alloc_read... done
[   15.446371] bcachefs (vdb): stripes_read... done
[   15.446376] bcachefs (vdb): snapshots_read... done
[   15.459256] bcachefs (vdb): journal_replay... done
[   15.460558] bcachefs (vdb): going read-write
[   15.466096] bcachefs (vdb): error mounting: error getting root inode: ENOENT_inode
[   15.468245] bcachefs (vdb): shutdown complete
[   15.468812] mount (858) used greatest stack depth: 11472 bytes left
[   15.470377] BUG: kernel NULL pointer dereference, address: 0000000000000018
[   15.471011] #PF: supervisor read access in kernel mode
[   15.471492] #PF: error_code(0x0000) - not-present page
[   15.471998] PGD 0 P4D 0 
[   15.472339] Oops: 0000 [#1] PREEMPT SMP PTI
[   15.472761] CPU: 0 PID: 1011 Comm: bch-rebalance/v Tainted: G           O       6.4.0-uksm+ #44 bb0e0ef3da5d44126bf2e48475ecd94d97cc8e12
[   15.473839] Hardware name: Alibaba Cloud Alibaba Cloud ECS, BIOS 449e491 04/01/2014
[   15.474442] RIP: 0010:bio_associate_blkg_from_css+0xcc/0x320
[   15.474948] Code: 4c 3b 38 0f 85 a0 00 00 00 e8 50 74 ae ff 49 8b 46 38 a8 03 75 5b 65 48 ff 00 e8 df af ae ff e8 da af ae ff eb 38 48 8b 45 08 <48> 8b 40 18 48 8b 98 38 01 00 00 e8 24 74 ae ff 48 8b 43 38 a8 03
[   15.476323] RSP: 0018:ffffc90000b0b448 EFLAGS: 00010246
[   15.476819] RAX: 0000000000000000 RBX: ffffffff82e2ba60 RCX: 0000000000000000
[   15.477411] RDX: ffff888102a74500 RSI: ffffffff82e2ba60 RDI: ffff88810d78a618
[   15.478009] RBP: ffff88810d78a618 R08: ffffea00040d7fc0 R09: 0000000000000000
[   15.478603] R10: ffff8881002ce400 R11: 0000000000000000 R12: 0000000000000000
[   15.479204] R13: ffff888105578800 R14: 0000000000000000 R15: ffff888107d4b800
[   15.479806] FS:  0000000000000000(0000) GS:ffff888237c00000(0000) knlGS:0000000000000000
[   15.480479] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   15.481031] CR2: 0000000000000018 CR3: 0000000002434005 CR4: 00000000003706f0
[   15.481665] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   15.482290] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   15.482892] Call Trace:
[   15.483248]  <TASK>
[   15.483584]  ? __die+0x1f/0x60
[   15.483975]  ? page_fault_oops+0x141/0x450
[   15.484407]  ? do_user_addr_fault+0x61/0x730
[   15.484856]  ? exc_page_fault+0x63/0x130
[   15.485296]  ? asm_exc_page_fault+0x22/0x30
[   15.485779]  ? bio_associate_blkg_from_css+0xcc/0x320
[   15.486285]  ? bio_add_page+0x24/0x60
[   15.486711]  bio_associate_blkg+0x28/0x60
[   15.487145]  bch2_btree_node_read+0x216/0x4b0 [bcachefs f5d54d621ddf1962c7405d16b44cd8fd013d4b32]
[   15.487864]  ? kvmalloc_node_noprof+0x3f/0xb0
[   15.488326]  bch2_btree_node_fill+0x2a0/0x530 [bcachefs f5d54d621ddf1962c7405d16b44cd8fd013d4b32]
[   15.489206]  bch2_btree_node_prefetch+0x61/0x80 [bcachefs f5d54d621ddf1962c7405d16b44cd8fd013d4b32]
[   15.490248]  btree_path_prefetch+0x20f/0x3e0 [bcachefs f5d54d621ddf1962c7405d16b44cd8fd013d4b32]
[   15.491246]  ? kvm_sched_clock_read+0x5/0x10
[   15.491969]  ? local_clock+0x9/0xd0
[   15.492614]  bch2_btree_path_traverse_one+0x52a/0xb50 [bcachefs f5d54d621ddf1962c7405d16b44cd8fd013d4b32]
[   15.493756]  ? __bch2_move_data+0x1c2/0x5e0 [bcachefs f5d54d621ddf1962c7405d16b44cd8fd013d4b32]
[   15.494818]  bch2_btree_iter_peek_upto+0x189/0xe30 [bcachefs f5d54d621ddf1962c7405d16b44cd8fd013d4b32]
[   15.495937]  ? kvm_clock_read+0x14/0x30
[   15.496629]  ? kvm_sched_clock_read+0x5/0x10
[   15.497359]  ? local_clock+0x9/0xd0
[   15.498040]  ? __bch2_move_data+0x1c2/0x5e0 [bcachefs f5d54d621ddf1962c7405d16b44cd8fd013d4b32]
[   15.499118]  __bch2_move_data+0x1c2/0x5e0 [bcachefs f5d54d621ddf1962c7405d16b44cd8fd013d4b32]
[   15.499931]  ? bch2_rebalance_thread+0x4e0/0x4e0 [bcachefs f5d54d621ddf1962c7405d16b44cd8fd013d4b32]
[   15.500663]  ? __bch2_move_data+0x135/0x5e0 [bcachefs f5d54d621ddf1962c7405d16b44cd8fd013d4b32]
[   15.501376]  ? __bch2_move_data+0x171/0x5e0 [bcachefs f5d54d621ddf1962c7405d16b44cd8fd013d4b32]
[   15.502112]  ? bch2_move_data+0xfa/0x1d0 [bcachefs f5d54d621ddf1962c7405d16b44cd8fd013d4b32]
[   15.502820]  bch2_move_data+0xfa/0x1d0 [bcachefs f5d54d621ddf1962c7405d16b44cd8fd013d4b32]
[   15.503503]  ? bch2_move_data+0x49/0x1d0 [bcachefs f5d54d621ddf1962c7405d16b44cd8fd013d4b32]
[   15.504203]  bch2_rebalance_thread+0x393/0x4e0 [bcachefs f5d54d621ddf1962c7405d16b44cd8fd013d4b32]
[   15.504944]  ? bch2_rebalance_thread+0x4e0/0x4e0 [bcachefs f5d54d621ddf1962c7405d16b44cd8fd013d4b32]
[   15.505675]  ? finish_task_switch.isra.0+0x91/0x290
[   15.506315]  ? rebalance_work+0x220/0x220 [bcachefs f5d54d621ddf1962c7405d16b44cd8fd013d4b32]
[   15.507242]  kthread+0xf0/0x120
[   15.507641]  ? kthread_complete_and_exit+0x20/0x20
[   15.508119]  ret_from_fork+0x1f/0x30
[   15.508527]  </TASK>
[   15.508863] Modules linked in: bcachefs(O) mean_and_variance sunrpc binfmt_misc nls_utf8 nls_cp437 intel_rapl_msr intel_rapl_common virtio_balloon virtio_console joydev evdev serio_raw squashfs loop dm_multipath dm_mod msr fuse efi_pstore ip_tables x_tables autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid1 raid0 multipath linear md_mod nvme_tcp nvme_rdma rdma_cm iw_cm ib_cm ib_core configfs nvme_fc nvme_fabrics crct10dif_pclmul crc32_pclmul ghash_clmulni_intel sha512_ssse3 virtio_net net_failover failover virtio_blk cirrus drm_shmem_helper aesni_intel crypto_simd drm_kms_helper cryptd psmouse virtio_pci drm virtio virtio_pci_legacy_dev i2c_piix4 virtio_pci_modern_dev virtio_ring i2c_core floppy pata_acpi button
[   15.513667] CR2: 0000000000000018
[   15.513669] ---[ end trace 0000000000000000 ]---
[   15.513671] RIP: 0010:bio_associate_blkg_from_css+0xcc/0x320
[   15.513677] Code: 4c 3b 38 0f 85 a0 00 00 00 e8 50 74 ae ff 49 8b 46 38 a8 03 75 5b 65 48 ff 00 e8 df af ae ff e8 da af ae ff eb 38 48 8b 45 08 <48> 8b 40 18 48 8b 98 38 01 00 00 e8 24 74 ae ff 48 8b 43 38 a8 03
[   15.513679] RSP: 0018:ffffc90000b0b448 EFLAGS: 00010246
[   15.513681] RAX: 0000000000000000 RBX: ffffffff82e2ba60 RCX: 0000000000000000
[   15.513682] RDX: ffff888102a74500 RSI: ffffffff82e2ba60 RDI: ffff88810d78a618
[   15.513684] RBP: ffff88810d78a618 R08: ffffea00040d7fc0 R09: 0000000000000000
[   15.513685] R10: ffff8881002ce400 R11: 0000000000000000 R12: 0000000000000000
[   15.513685] R13: ffff888105578800 R14: 0000000000000000 R15: ffff888107d4b800
[   15.513689] FS:  0000000000000000(0000) GS:ffff888237c00000(0000) knlGS:0000000000000000
[   15.513691] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   15.513692] CR2: 0000000000000018 CR3: 0000000002434005 CR4: 00000000003706f0
[   15.513693] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   15.513694] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   15.513695] note: bch-rebalance/v[1011] exited with irqs disabled
[   15.513866] bch-rebalance/v (1011) used greatest stack depth: 11168 bytes left
[   18.042919] loop6: detected capacity change from 0 to 8
jpsollie commented 1 year ago

same issue here. bcachefs fsck reported no errors, but only one of 3 filesystems could mount:

[   57.922615] bcachefs (8a4cc188-d598-4f75-8595-d53a560c60df): mounting version 1.1: snapshot_skiplists opts=metadata_replicas=3,data_checksum=xxhash,metadata_target=SSDM,foreground_target=SSDC,background_target=HDD,promote_target=SSDC
[   57.922698] bcachefs (8a4cc188-d598-4f75-8595-d53a560c60df): recovering from clean shutdown, journal seq 966443
[   57.937719] bcachefs (8a4cc188-d598-4f75-8595-d53a560c60df): alloc_read...
[   57.949996] ata5: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[   57.951777] ata5.00: configured for UDMA/133
[   57.982276]  done
[   57.982280] bcachefs (8a4cc188-d598-4f75-8595-d53a560c60df): stripes_read... done
[   57.982284] bcachefs (8a4cc188-d598-4f75-8595-d53a560c60df): snapshots_read... done
[   57.982297] bcachefs (8a4cc188-d598-4f75-8595-d53a560c60df): journal_replay... done
[   58.070303] bcachefs (8a4cc188-d598-4f75-8595-d53a560c60df): going read-write
[   58.082431] bcachefs (8a4cc188-d598-4f75-8595-d53a560c60df): error mounting: error getting root inode: ENOENT_inode
[   58.085976] BUG: kernel NULL pointer dereference, address: 0000000000000000
[   58.086519] #PF: supervisor read access in kernel mode
[   58.086852] #PF: error_code(0x0000) - not-present page
[   58.087163] PGD 0 P4D 0 
[   58.087465] Oops: 0000 [#1] SMP
[   58.087764] CPU: 0 PID: 3923 Comm: bch-rebalance/8 Not tainted 6.4.7release+ #1
[   58.088078] Hardware name: Gigabyte Technology Co., Ltd. X399 DESIGNARE EX/X399 DESIGNARE EX-CF, BIOS F13a 11/30/2021
[   58.088426] RIP: 0010:bch2_btree_node_read+0x3ce/0x430 [bcachefs]
[   58.088797] Code: fe ff ff 49 8b 47 70 a8 03 75 52 65 48 ff 08 31 d2 e9 44 fd ff ff 85 d2 0f 84 54 ff ff ff 8d 42 ff e9 43 ff ff ff 49 8b 57 78 <48> 8b 0a 48 85 c9 74 0e 48 8d 71 01 48 89 c8 f0 48 0f b1 32 75 35
[   58.089533] RSP: 0018:ffffc90000d4b438 EFLAGS: 00010206
[   58.089910] RAX: 0000000000000007 RBX: ffff8883c0f08258 RCX: 000000000000001f
[   58.090299] RDX: 0000000000000000 RSI: 0000000024924a83 RDI: ffffffd75f04f4f0
[   58.090683] RBP: ffffc90000d4b4d8 R08: 0000000000000002 R09: 000000000000004a
[   58.091071] R10: 0000000000000001 R11: 99961f7237728456 R12: ffff88840d400000
[   58.091460] R13: ffff889ffeaae800 R14: 0000000000000000 R15: ffff88844d181800
[   58.091849] FS:  0000000000000000(0000) GS:ffff889ffe800000(0000) knlGS:0000000000000000
[   58.092238] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   58.092618] CR2: 0000000000000000 CR3: 00000001c2395000 CR4: 00000000003506f0
[   58.092924] Call Trace:
[   58.093224]  <TASK>
[   58.093525]  ? __die+0x28/0x80
[   58.093829]  ? page_fault_oops+0x15c/0x430
[   58.094131]  ? __rq_qos_issue+0x1c/0x30
[   58.094431]  ? blk_mq_start_request+0x25/0x90
[   58.094728]  ? nvme_queue_rq+0xbd/0x230
[   58.095026]  ? ipi_sync_rq_state+0x30/0x30
[   58.095327]  ? exc_page_fault+0x3ba/0x5e0
[   58.095623]  ? kmem_cache_alloc_noprof+0x23/0x330
[   58.095919]  ? asm_exc_page_fault+0x22/0x30
[   58.096210]  ? bch2_btree_node_read+0x3ce/0x430 [bcachefs]
[   58.096520]  ? bch2_btree_node_read+0xd7/0x430 [bcachefs]
[   58.096830]  ? bch2_btree_node_mem_alloc+0x23c/0x5e0 [bcachefs]
[   58.097137]  ? bch2_btree_node_fill+0x221/0x3d0 [bcachefs]
[   58.097448]  ? bch2_btree_node_prefetch+0x8b/0xb0 [bcachefs]
[   58.097757]  ? btree_path_prefetch+0x181/0x330 [bcachefs]
[   58.098071]  ? bch2_bkey_unpack+0x31/0xd0 [bcachefs]
[   58.098386]  ? bch2_btree_path_traverse_one+0x3f8/0x8f0 [bcachefs]
[   58.098708]  ? __bch2_move_data+0x215/0x5a0 [bcachefs]
[   58.099027]  ? __bch2_btree_path_set_pos+0x3a5/0x430 [bcachefs]
[   58.099346]  ? bch2_btree_iter_peek_upto+0x163/0xc20 [bcachefs]
[   58.099667]  ? bch2_moving_ctxt_do_pending_writes+0x28/0x140 [bcachefs]
[   58.099990]  ? local_clock+0x14/0xd0
[   58.100296]  ? bch2_trans_begin+0x2c6/0x560 [bcachefs]
[   58.100628]  ? __bch2_move_data+0x215/0x5a0 [bcachefs]
[   58.100957]  ? bch2_rebalance_thread+0x4d0/0x4d0 [bcachefs]
[   58.101288]  ? __bch2_move_data+0x1c1/0x5a0 [bcachefs]
[   58.101621]  ? bch2_move_data+0xf1/0x1c0 [bcachefs]
[   58.101950]  ? bch2_move_data+0x44/0x1c0 [bcachefs]
[   58.102271]  ? bch2_rebalance_thread+0x3b7/0x4d0 [bcachefs]
[   58.102593]  ? bch2_rebalance_thread+0x4d0/0x4d0 [bcachefs]
[   58.102907]  ? __schedule+0x289/0xfc0
[   58.103207]  ? rebalance_work+0x1e0/0x1e0 [bcachefs]
[   58.103517]  ? kthread+0xbe/0xf0
[   58.103799]  ? kthread_exit+0x30/0x30
[   58.104066]  ? ret_from_fork+0x1f/0x30
[   58.104321]  </TASK>
[   58.104603] Modules linked in: bcachefs lz4hc_compress mean_and_variance i2c_dev amdgpu intel_rapl_msr intel_rapl_common iosf_mbi kvm_amd drm_ttm_helper ttm gpu_sched kvm drm_buddy iwlmvm drm_suballoc_helper drm_display_helper mac80211 drm_kms_helper irqbypass binfmt_misc libarc4 crct10dif_pclmul syscopyarea polyval_clmulni sysfillrect polyval_generic sysimgblt btusb sha512_ssse3 iwlwifi atlantic btintel rapl wmi_bmof gigabyte_wmi hid_generic bluetooth i2c_piix4 mpt3sas ccp gpio_amdpt gpio_generic efivarfs
[   58.105991] CR2: 0000000000000000
[   58.106344] ---[ end trace 0000000000000000 ]---
[   58.106345] BUG: kernel NULL pointer dereference, address: 0000000000000000
[   58.107032] #PF: supervisor write access in kernel mode
[   58.107350] #PF: error_code(0x0002) - not-present page
[   58.107675] PGD 0 P4D 0 
[   58.108006] Oops: 0002 [#2] SMP
[   58.108317] CPU: 17 PID: 239 Comm: kworker/17:1 Tainted: G      D            6.4.7release+ #1
[   58.109236] Hardware name: Gigabyte Technology Co., Ltd. X399 DESIGNARE EX/X399 DESIGNARE EX-CF, BIOS F13a 11/30/2021
[   58.124170] Workqueue: bcachefs_write_ref bch2_do_invalidates_work [bcachefs]
[   58.124533] RIP: 0010:bch2_do_invalidates_work+0x9f0/0xac0 [bcachefs]
[   58.124896] Code: 6a fe ff ff 49 8d b6 a4 01 00 00 48 c7 c7 88 f9 3f a0 48 89 54 24 30 e8 ae 11 9e e0 48 8b 54 24 30 e9 1a ff ff ff 49 8b 45 48 <f0> 48 83 28 01 0f 85 81 fd ff ff 49 8b 45 48 49 8d 7d 40 ff 50 08
[   58.125625] RSP: 0018:ffffc90000a47ac8 EFLAGS: 00010206
[   58.141074] RAX: 0000000000000000 RBX: 0000000000000004 RCX: 0000000000000000
[   58.141455] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000022
[   58.141831] RBP: ffffc90000a47e90 R08: 0000000000000000 R09: 0003ffffffffffff
[   58.142207] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000003
[   58.143273] R13: ffff88844d180000 R14: ffff88840d400000 R15: 0000ffffffffffff
[   58.158835] FS:  0000000000000000(0000) GS:ffff889ffec40000(0000) knlGS:0000000000000000
[   58.159223] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   58.159628] CR2: 0000000000000000 CR3: 0000000072e0e000 CR4: 00000000003506e0
[   58.175037] Call Trace:
[   58.175501]  <TASK>
[   58.188864]  ? __die+0x28/0x80
[   58.189279]  ? page_fault_oops+0x15c/0x430
[   58.189696]  ? exc_page_fault+0x3ba/0x5e0
[   58.190702]  ? bch2_btree_iter_peek_upto+0x1f0/0xc20 [bcachefs]
[   58.202350]  ? __bch2_btree_path_set_pos+0x3a5/0x430 [bcachefs]
[   58.202750]  ? asm_exc_page_fault+0x22/0x30
[   58.203130]  ? bch2_do_invalidates_work+0x9f0/0xac0 [bcachefs]
[   58.203531]  ? bch2_do_invalidates_work+0x76c/0xac0 [bcachefs]
[   58.219384]  ? load_balance+0x245/0xbb0
[   58.219809]  ? bch2_do_invalidates_work+0x1f0/0xac0 [bcachefs]
[   58.220207]  ? process_one_work+0x1b6/0x330
[   58.220591]  ? worker_thread+0x45/0x3d0
[   58.232613]  ? process_one_work+0x330/0x330
[   58.233001]  ? kthread+0xbe/0xf0
[   58.233386]  ? kthread_exit+0x30/0x30
[   58.233767]  ? ret_from_fork+0x1f/0x30
[   58.234145]  </TASK>
[   58.248337] Modules linked in: bcachefs lz4hc_compress mean_and_variance i2c_dev amdgpu intel_rapl_msr intel_rapl_common iosf_mbi kvm_amd drm_ttm_helper ttm gpu_sched kvm drm_buddy iwlmvm drm_suballoc_helper drm_display_helper mac80211 drm_kms_helper irqbypass binfmt_misc libarc4 crct10dif_pclmul syscopyarea polyval_clmulni sysfillrect polyval_generic sysimgblt btusb sha512_ssse3 iwlwifi atlantic btintel rapl wmi_bmof gigabyte_wmi hid_generic bluetooth i2c_piix4 mpt3sas ccp gpio_amdpt gpio_generic efivarfs
[   58.265281] CR2: 0000000000000000
[   58.265687] ---[ end trace 0000000000000000 ]---
[   58.406701] RIP: 0010:bch2_btree_node_read+0x3ce/0x430 [bcachefs]
[   58.407130] Code: fe ff ff 49 8b 47 70 a8 03 75 52 65 48 ff 08 31 d2 e9 44 fd ff ff 85 d2 0f 84 54 ff ff ff 8d 42 ff e9 43 ff ff ff 49 8b 57 78 <48> 8b 0a 48 85 c9 74 0e 48 8d 71 01 48 89 c8 f0 48 0f b1 32 75 35
[   58.407948] RSP: 0018:ffffc90000d4b438 EFLAGS: 00010206
[   58.408357] RAX: 0000000000000007 RBX: ffff8883c0f08258 RCX: 000000000000001f
[   58.423977] RDX: 0000000000000000 RSI: 0000000024924a83 RDI: ffffffd75f04f4f0
[   58.424392] RBP: ffffc90000d4b4d8 R08: 0000000000000002 R09: 000000000000004a
[   58.424808] R10: 0000000000000001 R11: 99961f7237728456 R12: ffff88840d400000
[   58.425224] R13: ffff889ffeaae800 R14: 0000000000000000 R15: ffff88844d181800
[   58.442717] FS:  0000000000000000(0000) GS:ffff889ffe800000(0000) knlGS:0000000000000000
[   58.443127] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   58.443530] CR2: 0000000000000000 CR3: 00000001c2395000 CR4: 00000000003506f0
[   58.443930] note: bch-rebalance/8[3923] exited with irqs disabled
[   58.690357] RIP: 0010:bch2_btree_node_read+0x3ce/0x430 [bcachefs]
[   58.690852] Code: fe ff ff 49 8b 47 70 a8 03 75 52 65 48 ff 08 31 d2 e9 44 fd ff ff 85 d2 0f 84 54 ff ff ff 8d 42 ff e9 43 ff ff ff 49 8b 57 78 <48> 8b 0a 48 85 c9 74 0e 48 8d 71 01 48 89 c8 f0 48 0f b1 32 75 35
[   58.691677] RSP: 0018:ffffc90000d4b438 EFLAGS: 00010206
[   58.692106] RAX: 0000000000000007 RBX: ffff8883c0f08258 RCX: 000000000000001f
[   58.692527] RDX: 0000000000000000 RSI: 0000000024924a83 RDI: ffffffd75f04f4f0
[   58.692947] RBP: ffffc90000d4b4d8 R08: 0000000000000002 R09: 000000000000004a
[   58.693363] R10: 0000000000000001 R11: 99961f7237728456 R12: ffff88840d400000
[   58.693777] R13: ffff889ffeaae800 R14: 0000000000000000 R15: ffff88844d181800
[   58.694190] FS:  0000000000000000(0000) GS:ffff889ffec40000(0000) knlGS:0000000000000000
[   58.694607] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   58.695026] CR2: 0000000000000000 CR3: 000000018ece5000 CR4: 00000000003506e0
[   58.695451] note: kworker/17:1[239] exited with irqs disabled
[   58.723596] bcachefs (58671ce3-7ba5-43a8-ad57-70ccef8d968e): mounting version 1.1: snapshot_skiplists opts=metadata_replicas=2,data_checksum=xxhash,metadata_target=SSDM,foreground_target=SSDC,background_target=HDD,promote_target=SSDC
[   58.725046] bcachefs (58671ce3-7ba5-43a8-ad57-70ccef8d968e): recovering from clean shutdown, journal seq 700017
[   58.787835] bcachefs (58671ce3-7ba5-43a8-ad57-70ccef8d968e): alloc_read... done
[   58.801903] bcachefs (58671ce3-7ba5-43a8-ad57-70ccef8d968e): stripes_read... done
[   58.802336] bcachefs (58671ce3-7ba5-43a8-ad57-70ccef8d968e): snapshots_read... done
[   58.802775] bcachefs (58671ce3-7ba5-43a8-ad57-70ccef8d968e): journal_replay... done
[   58.866146] bcachefs (58671ce3-7ba5-43a8-ad57-70ccef8d968e): going read-write
[   58.921793] bcachefs (60f92ebb-8d42-4536-a36f-8e6621518376): mounting version 1.1: snapshot_skiplists opts=metadata_replicas=2,metadata_checksum=xxhash,data_checksum=xxhash,metadata_target=SSDM,foreground_target=SSDC,background_target=HDD,promote_target=SSDC
[   58.923390] bcachefs (60f92ebb-8d42-4536-a36f-8e6621518376): recovering from clean shutdown, journal seq 58845
[   58.947025] bcachefs (60f92ebb-8d42-4536-a36f-8e6621518376): alloc_read... done
[   58.952256] bcachefs (60f92ebb-8d42-4536-a36f-8e6621518376): stripes_read... done
[   58.953319] bcachefs (60f92ebb-8d42-4536-a36f-8e6621518376): snapshots_read... done
[   58.954437] bcachefs (60f92ebb-8d42-4536-a36f-8e6621518376): journal_replay... done
[   58.975227] bcachefs (60f92ebb-8d42-4536-a36f-8e6621518376): going read-write
[   58.982185] bcachefs (60f92ebb-8d42-4536-a36f-8e6621518376): error mounting: error getting root inode: ENOENT_inode
[   58.986150] BUG: kernel NULL pointer dereference, address: 0000000000000000
[   58.986978] #PF: supervisor read access in kernel mode
[   58.987664] #PF: error_code(0x0000) - not-present page
[   58.988136] PGD 0 P4D 0 
[   58.988598] Oops: 0000 [#3] SMP
[   58.989039] CPU: 8 PID: 3952 Comm: bch-rebalance/6 Tainted: G      D            6.4.7release+ #1
[   58.989472] Hardware name: Gigabyte Technology Co., Ltd. X399 DESIGNARE EX/X399 DESIGNARE EX-CF, BIOS F13a 11/30/2021
[   58.989903] RIP: 0010:bch2_btree_node_read+0x3ce/0x430 [bcachefs]
[   58.990343] Code: fe ff ff 49 8b 47 70 a8 03 75 52 65 48 ff 08 31 d2 e9 44 fd ff ff 85 d2 0f 84 54 ff ff ff 8d 42 ff e9 43 ff ff ff 49 8b 57 78 <48> 8b 0a 48 85 c9 74 0e 48 8d 71 01 48 89 c8 f0 48 0f b1 32 75 35
[   58.991171] RSP: 0018:ffffc900016d7438 EFLAGS: 00010206
[   58.991581] RAX: 0000000000000007 RBX: ffff88844d9100d8 RCX: 000000000000001f
[   58.991994] RDX: 0000000000000000 RSI: 0000000024924a83 RDI: ffffffd75f04f4f0
[   58.992400] RBP: ffffc900016d74d8 R08: 0000000000000002 R09: 0000000000000004
[   58.992801] R10: 0000000000023478 R11: ffffffff822aa8c0 R12: ffff8883cef40000
[   58.993201] R13: ffff8882115c9600 R14: 0000000000000000 R15: ffff88810b54d000
[   58.993600] FS:  0000000000000000(0000) GS:ffff889ffea00000(0000) knlGS:0000000000000000
[   58.994002] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   58.994400] CR2: 0000000000000000 CR3: 00000003417f4000 CR4: 00000000003506e0
[   58.994794] Call Trace:
[   58.995175]  <TASK>
[   58.995548]  ? __die+0x28/0x80
[   58.995920]  ? page_fault_oops+0x15c/0x430
[   58.996291]  ? exc_page_fault+0x3ba/0x5e0
[   58.996658]  ? kmem_cache_alloc_noprof+0x2df/0x330
[   58.997026]  ? asm_exc_page_fault+0x22/0x30
[   58.997390]  ? bch2_btree_node_read+0x3ce/0x430 [bcachefs]
[   58.997769]  ? bch2_btree_node_read+0xd7/0x430 [bcachefs]
[   58.998143]  ? bch2_btree_node_mem_alloc+0x23c/0x5e0 [bcachefs]
[   58.998513]  ? bch2_btree_node_fill+0x221/0x3d0 [bcachefs]
[   58.998878]  ? bch2_btree_node_prefetch+0x8b/0xb0 [bcachefs]
[   58.999234]  ? btree_path_prefetch+0x181/0x330 [bcachefs]
[   58.999581]  ? bch2_btree_path_traverse_one+0x3f8/0x8f0 [bcachefs]
[   58.999923]  ? __bch2_move_data+0x215/0x5a0 [bcachefs]
[   59.000266]  ? bch2_btree_iter_peek_upto+0x163/0xc20 [bcachefs]
[   59.000611]  ? bch2_moving_ctxt_do_pending_writes+0x28/0x140 [bcachefs]
[   59.000961]  ? local_clock+0x14/0xd0
[   59.001293]  ? bch2_trans_begin+0x2c6/0x560 [bcachefs]
[   59.001648]  ? __bch2_move_data+0x215/0x5a0 [bcachefs]
[   59.001993]  ? bch2_rebalance_thread+0x4d0/0x4d0 [bcachefs]
[   59.002330]  ? __bch2_move_data+0x1c1/0x5a0 [bcachefs]
[   59.002657]  ? bch2_move_data+0xf1/0x1c0 [bcachefs]
[   59.002982]  ? bch2_move_data+0x44/0x1c0 [bcachefs]
[   59.003306]  ? bch2_rebalance_thread+0x3b7/0x4d0 [bcachefs]
[   59.003631]  ? bch2_rebalance_thread+0x4d0/0x4d0 [bcachefs]
[   59.003955]  ? __schedule+0x289/0xfc0
[   59.004262]  ? rebalance_work+0x1e0/0x1e0 [bcachefs]
[   59.004585]  ? kthread+0xbe/0xf0
[   59.004882]  ? kthread_exit+0x30/0x30
[   59.005175]  ? ret_from_fork+0x1f/0x30
[   59.005465]  </TASK>
[   59.005749] Modules linked in: bcachefs lz4hc_compress mean_and_variance i2c_dev amdgpu intel_rapl_msr intel_rapl_common iosf_mbi kvm_amd drm_ttm_helper ttm gpu_sched kvm drm_buddy iwlmvm drm_suballoc_helper drm_display_helper mac80211 drm_kms_helper irqbypass binfmt_misc libarc4 crct10dif_pclmul syscopyarea polyval_clmulni sysfillrect polyval_generic sysimgblt btusb sha512_ssse3 iwlwifi atlantic btintel rapl wmi_bmof gigabyte_wmi hid_generic bluetooth i2c_piix4 mpt3sas ccp gpio_amdpt gpio_generic efivarfs
[   59.007022] CR2: 0000000000000000
[   59.007331] ---[ end trace 0000000000000000 ]---
[   59.007331] BUG: kernel NULL pointer dereference, address: 0000000000000000
[   59.007333] #PF: supervisor read access in kernel mode
[   59.008596] #PF: error_code(0x0000) - not-present page
[   59.009075] PGD 0 P4D 0 
[   59.009547] Oops: 0000 [#4] SMP
[   59.010014] CPU: 24 PID: 3491 Comm: kworker/24:2 Tainted: G      D            6.4.7release+ #1
[   59.026224] Hardware name: Gigabyte Technology Co., Ltd. X399 DESIGNARE EX/X399 DESIGNARE EX-CF, BIOS F13a 11/30/2021
[   59.026718] Workqueue: bcachefs_write_ref bch2_do_invalidates_work [bcachefs]
[   59.027247] RIP: 0010:bch2_btree_node_read+0x3ce/0x430 [bcachefs]
[   59.027763] Code: fe ff ff 49 8b 47 70 a8 03 75 52 65 48 ff 08 31 d2 e9 44 fd ff ff 85 d2 0f 84 54 ff ff ff 8d 42 ff e9 43 ff ff ff 49 8b 57 78 <48> 8b 0a 48 85 c9 74 0e 48 8d 71 01 48 89 c8 f0 48 0f b1 32 75 35
[   59.028793] RSP: 0018:ffffc900014377a8 EFLAGS: 00010206
[   59.045263] RAX: 0000000000000007 RBX: ffff888103c64258 RCX: 000000000000001f
[   59.045807] RDX: 0000000000000000 RSI: 0000000024924a83 RDI: ffffffd75f04f4f0
[   59.046351] RBP: ffffc90001437848 R08: 0000000000000002 R09: 000000006dde172f
[   59.046897] R10: 0000000000000001 R11: 000000003f03ee4e R12: ffff8883cef40000
[   59.047446] R13: ffff8882115c9800 R14: 0000000000000001 R15: ffff88810b54e800
[   59.061043] FS:  0000000000000000(0000) GS:ffff889ffee00000(0000) knlGS:0000000000000000
[   59.061607] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   59.062169] CR2: 0000000000000000 CR3: 0000000072e0e000 CR4: 00000000003506e0
[   59.062739] Call Trace:
[   59.077267]  <TASK>
[   59.077827]  ? __die+0x28/0x80
[   59.078386]  ? page_fault_oops+0x15c/0x430
[   59.078941]  ? __bch2_time_stats_update+0x109/0x250 [bcachefs]
[   59.079521]  ? exc_page_fault+0x3ba/0x5e0
[   59.096295]  ? kmem_cache_alloc_noprof+0x23/0x330
[   59.096857]  ? asm_exc_page_fault+0x22/0x30
[   59.097408]  ? bch2_btree_node_read+0x3ce/0x430 [bcachefs]
[   59.097988]  ? bch2_btree_node_read+0xd7/0x430 [bcachefs]
[   59.098564]  ? bch2_btree_node_mem_alloc+0x23c/0x5e0 [bcachefs]
[   59.114351]  ? bch2_btree_node_fill+0x2df/0x3d0 [bcachefs]
[   59.114939]  ? __bch2_btree_node_get.isra.0+0x283/0x500 [bcachefs]
[   59.115522]  ? bch2_btree_path_traverse_one+0x1f3/0x8f0 [bcachefs]
[   59.116119]  ? bch2_do_invalidates_work+0x24e/0xac0 [bcachefs]
[   59.131747]  ? bch2_btree_iter_peek_upto+0x163/0xc20 [bcachefs]
[   59.132339]  ? local_clock+0x14/0xd0
[   59.132902]  ? bch2_trans_begin+0x2c6/0x560 [bcachefs]
[   59.133492]  ? bch2_do_invalidates_work+0xbb/0xac0 [bcachefs]
[   59.135482]  ? bch2_do_invalidates_work+0x24e/0xac0 [bcachefs]
[   59.151219]  ? bch2_do_invalidates_work+0x1f0/0xac0 [bcachefs]
[   59.151876]  ? process_one_work+0x1b6/0x330
[   59.152439]  ? worker_thread+0x45/0x3d0
[   59.152982]  ? process_one_work+0x330/0x330
[   59.169456]  ? kthread+0xbe/0xf0
[   59.169964]  ? kthread_exit+0x30/0x30
[   59.170451]  ? ret_from_fork+0x1f/0x30
[   59.170935]  </TASK>
[   59.171407] Modules linked in: bcachefs lz4hc_compress mean_and_variance i2c_dev amdgpu intel_rapl_msr intel_rapl_common iosf_mbi kvm_amd drm_ttm_helper ttm gpu_sched kvm drm_buddy iwlmvm drm_suballoc_helper drm_display_helper mac80211 drm_kms_helper irqbypass binfmt_misc libarc4 crct10dif_pclmul syscopyarea polyval_clmulni sysfillrect polyval_generic sysimgblt btusb sha512_ssse3 iwlwifi atlantic btintel rapl wmi_bmof gigabyte_wmi hid_generic bluetooth i2c_piix4 mpt3sas ccp gpio_amdpt gpio_generic efivarfs
[   59.189438] CR2: 0000000000000000
[   59.203781] ---[ end trace 0000000000000000 ]---
[   59.297923] RIP: 0010:bch2_btree_node_read+0x3ce/0x430 [bcachefs]
[   59.298489] Code: fe ff ff 49 8b 47 70 a8 03 75 52 65 48 ff 08 31 d2 e9 44 fd ff ff 85 d2 0f 84 54 ff ff ff 8d 42 ff e9 43 ff ff ff 49 8b 57 78 <48> 8b 0a 48 85 c9 74 0e 48 8d 71 01 48 89 c8 f0 48 0f b1 32 75 35
[   59.299563] RSP: 0018:ffffc90000d4b438 EFLAGS: 00010206
[   59.300086] RAX: 0000000000000007 RBX: ffff8883c0f08258 RCX: 000000000000001f
[   59.300609] RDX: 0000000000000000 RSI: 0000000024924a83 RDI: ffffffd75f04f4f0
[   59.316645] RBP: ffffc90000d4b4d8 R08: 0000000000000002 R09: 000000000000004a
[   59.317166] R10: 0000000000000001 R11: 99961f7237728456 R12: ffff88840d400000
[   59.317687] R13: ffff889ffeaae800 R14: 0000000000000000 R15: ffff88844d181800
[   59.318207] FS:  0000000000000000(0000) GS:ffff889ffea00000(0000) knlGS:0000000000000000
[   59.318736] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   59.336255] CR2: 0000000000000000 CR3: 00000003417f4000 CR4: 00000000003506e0
[   59.336782] note: bch-rebalance/6[3952] exited with irqs disabled
[   59.572317] RIP: 0010:bch2_btree_node_read+0x3ce/0x430 [bcachefs]
[   59.572907] Code: fe ff ff 49 8b 47 70 a8 03 75 52 65 48 ff 08 31 d2 e9 44 fd ff ff 85 d2 0f 84 54 ff ff ff 8d 42 ff e9 43 ff ff ff 49 8b 57 78 <48> 8b 0a 48 85 c9 74 0e 48 8d 71 01 48 89 c8 f0 48 0f b1 32 75 35
[   59.573828] RSP: 0018:ffffc90000d4b438 EFLAGS: 00010206
[   59.574244] RAX: 0000000000000007 RBX: ffff8883c0f08258 RCX: 000000000000001f
[   59.574659] RDX: 0000000000000000 RSI: 0000000024924a83 RDI: ffffffd75f04f4f0
[   59.575063] RBP: ffffc90000d4b4d8 R08: 0000000000000002 R09: 000000000000004a
[   59.575466] R10: 0000000000000001 R11: 99961f7237728456 R12: ffff88840d400000
[   59.575804] R13: ffff889ffeaae800 R14: 0000000000000000 R15: ffff88844d181800
[   59.576051] FS:  0000000000000000(0000) GS:ffff889ffee00000(0000) knlGS:0000000000000000
[   59.576302] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   59.576552] CR2: 0000000000000000 CR3: 00000003417f4000 CR4: 00000000003506e0
[   59.576806] note: kworker/24:2[3491] exited with irqs disabled

... and this one got mounted:

bcachefs show-super /dev/md3
External UUID:                              58671ce3-7ba5-43a8-ad57-70ccef8d968e
Internal UUID:                              13cdc6d5-ecdc-439b-aedb-17420e6003ba
koverstreet commented 1 year ago

pipe it through scripts/decode_stacktrace.sh

jpsollie commented 1 year ago

When I'm using frame pointer unwinder, it's not capable of decoding it. May it be related to compiled as a module?

jpsollie commented 1 year ago

stacktrace.txt

koverstreet commented 1 year ago

You gave it the wrong vmlinux - notice how it only has question marks, not file/line numbers.

jpsollie commented 1 year ago

yes, so it seems, so I recompiled everything, verified with uname -a that the kernel which I booted was indeed the one at /usr/src/linux/vmlinux (even tried with -r 6.4.7release+), but it didn't succeed. Is there anything in my .config which causes this? config.txt

EDIT updated to e044e6e4, enabled preemptible kernel & ftrace, but decode_stacktrace still shows pretty limited output stacktrace.txt

jpsollie commented 1 year ago

@koverstreet : in meantime, I already reformatted 2 instances, they are working flawlessly. I assume it has something to do with the upgrade process having issues with snapshots Would you like a bcachefs metadata export before I reformat the remaining one?

koverstreet commented 1 year ago

Yes, please

jpsollie commented 1 year ago

sent. I'm unsubscribing now as no longer encountering this issue