search

koverstreet / bcachefs

Other
643 stars 71 forks source link

running the mongodb with snapshots kernel panic #607

Closed bhzhu203 closed 8 months ago

bhzhu203 commented 8 months ago

When run the mongodb with snapshot for minutes , the kernel panic immediately ,version 829f77e7fe08fec96234bbd38faf62854712777f .

There is good on vesion 55c11a159d3ca4ca7f9d5c1275d0768474b12195

[  364.983159] ------------[ cut here ]------------
[  364.983501] kernel BUG at fs/bcachefs/btree_trans_commit.c:100!
[  364.983844] invalid opcode: 0000 [#1] PREEMPT SMP PTI
[  364.984134] CPU: 3 PID: 87 Comm: kworker/u8:4 Kdump: loaded Tainted: G           O       6.6.0-uksm+ #5 9493d6624cad7096f2176856d98788fd783d4179
[  364.984847] Hardware name: Alibaba Cloud Alibaba Cloud ECS, BIOS 449e491 04/01/2014
[  364.985289] Workqueue: bcachefs bch2_write_point_do_index_updates [bcachefs]
[  364.985722] RIP: 0010:bch2_btree_bset_insert_key+0x406/0x620 [bcachefs]
[  364.986113] Code: e9 d1 fe ff ff 0f 0b 0f 0b 0f 0b 48 8b 74 24 08 44 89 cf 48 33 56 18 4c 33 56 10 4c 09 d2 33 7e 0c 48 09 fa 0f 84 45 fd ff ff <0f> 0b 66 83 ab d0 00 00 00 01 e9 3d ff ff ff 66 83 ab d0 00 00 00
[  364.987119] RSP: 0018:ffffc900010038e0 EFLAGS: 00010206
[  364.987417] RAX: ffff88810f340000 RBX: ffff8881042f5a00 RCX: ffff8881042f5ab2
[  364.987820] RDX: 0000000000000006 RSI: ffff88810f1203f0 RDI: 0000000000000006
[  364.988219] RBP: ffff888106a580c0 R08: 0000000000000000 R09: 00000000fffffffd
[  364.988623] R10: 0000000000000000 R11: 00000000000000c5 R12: ffff88810f120420
[  364.989023] R13: ffff88810f120000 R14: ffff8881042f5aa8 R15: 0000000000000000
[  364.989428] FS:  0000000000000000(0000) GS:ffff888237d80000(0000) knlGS:0000000000000000
[  364.989877] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  364.990217] CR2: 0000563c804d8040 CR3: 000000016c088004 CR4: 00000000003706e0
[  364.990620] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  364.991028] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  364.991429] Call Trace:
[  364.991579]  <TASK>
[  364.991711]  ? die+0x32/0x80
[  364.991890]  ? do_trap+0xd2/0x100
[  364.992087]  ? bch2_btree_bset_insert_key+0x406/0x620 [bcachefs b6f528eddb51be4621e115756aa823e663f235ad]
[  364.992637]  ? do_error_trap+0x65/0x80
[  364.992855]  ? bch2_btree_bset_insert_key+0x406/0x620 [bcachefs b6f528eddb51be4621e115756aa823e663f235ad]
[  364.993400]  ? exc_invalid_op+0x49/0x60
[  364.993624]  ? bch2_btree_bset_insert_key+0x406/0x620 [bcachefs b6f528eddb51be4621e115756aa823e663f235ad]
[  364.994164]  ? asm_exc_invalid_op+0x16/0x20
[  364.994412]  ? bch2_btree_bset_insert_key+0x406/0x620 [bcachefs b6f528eddb51be4621e115756aa823e663f235ad]
[  364.994956]  ? bch2_btree_path_verify+0x20/0xd0 [bcachefs b6f528eddb51be4621e115756aa823e663f235ad]
[  364.995490]  ? __bch2_btree_path_set_pos+0x1c9/0x4f0 [bcachefs b6f528eddb51be4621e115756aa823e663f235ad]
[  364.996029]  ? __bch2_btree_write_buffer_flush+0x355/0xbc0 [bcachefs b6f528eddb51be4621e115756aa823e663f235ad]
[  364.996604]  bch2_btree_insert_key_leaf+0x8e/0x280 [bcachefs b6f528eddb51be4621e115756aa823e663f235ad]
[  364.997136]  __bch2_btree_write_buffer_flush+0x55f/0xbc0 [bcachefs b6f528eddb51be4621e115756aa823e663f235ad]
[  364.997699]  ? __bch2_btree_write_buffer_flush+0x355/0xbc0 [bcachefs b6f528eddb51be4621e115756aa823e663f235ad]
[  364.998268]  ? __bch2_trans_commit+0x678/0x1d80 [bcachefs b6f528eddb51be4621e115756aa823e663f235ad]
[  364.998787]  __bch2_trans_commit+0x678/0x1d80 [bcachefs b6f528eddb51be4621e115756aa823e663f235ad]
[  364.999297]  bch2_extent_update+0x171/0x230 [bcachefs b6f528eddb51be4621e115756aa823e663f235ad]
[  364.999810]  bch2_write_index_default+0x2db/0x430 [bcachefs b6f528eddb51be4621e115756aa823e663f235ad]
[  365.000363]  ? bch2_write_index_default+0x10f/0x430 [bcachefs b6f528eddb51be4621e115756aa823e663f235ad]
[  365.000909]  ? __bch2_write_index+0x191/0x260 [bcachefs b6f528eddb51be4621e115756aa823e663f235ad]
[  365.001425]  __bch2_write_index+0x191/0x260 [bcachefs b6f528eddb51be4621e115756aa823e663f235ad]
[  365.002245]  bch2_write_point_do_index_updates+0x9d/0x140 [bcachefs b6f528eddb51be4621e115756aa823e663f235ad]
[  365.003104]  process_one_work+0x133/0x2f0
[  365.003598]  worker_thread+0x2ed/0x410
[  365.004081]  ? rescuer_thread+0x410/0x410
[  365.004628]  kthread+0xf0/0x120
[  365.005062]  ? kthread_complete_and_exit+0x20/0x20
[  365.005619]  ret_from_fork+0x2d/0x50
[  365.006063]  ? kthread_complete_and_exit+0x20/0x20
[  365.006560]  ret_from_fork_asm+0x11/0x20
[  365.007008]  </TASK>
[  365.007367] Modules linked in: netconsole bcachefs(O) mean_and_variance tcp_diag inet_diag sunrpc binfmt_misc nls_utf8 nls_cp437 intel_rapl_msr intel_rapl_common joydev serio_raw virtio_console virtio_balloon evdev squashfs loop dm_multipath dm_mod msr fuse efi_pstore ip_tables x_tables autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid1 raid0 multipath linear md_mod nvme_tcp nvme_rdma rdma_cm iw_cm ib_cm ib_core configfs nvme_fc nvme_fabrics crct10dif_pclmul crc32_pclmul ghash_clmulni_intel sha512_ssse3 virtio_net net_failover failover virtio_blk cirrus drm_shmem_helper aesni_intel drm_kms_helper crypto_simd floppy cryptd virtio_pci psmouse drm virtio i2c_piix4 virtio_pci_legacy_dev virtio_pci_modern_dev i2c_core virtio_ring pata_acpi button

❰bhzhu❙~/source/bcachfs-github/bcachefs(git:master)❱✔≻                                                                       bch2_btree_bset_insert_key+0x406/0x620
bch2_btree_bset_insert_key+0x406/0x620:s.ko  bch2_btree_bset_insert_key+0x406/0x620
bch2_btree_bset_insert_key 于 /home/bhzhu/source/bcachfs-github/bcachefs/fs/bcachefs/btree_trans_commit.c:100 (discriminator 3)
❰bhzhu❙~/source/bcachfs-github/bcachefs(git:master)❱✔≻ export LANG=C
❰bhzhu❙~/source/bcachfs-github/bcachefs(git:master)❱✔≻ scripts/faddr2line  fs/bcachefs/bcachefs.ko  bch2_btree_bset_insert_key+0x406/0x620
bch2_btree_bset_insert_key+0x406/0x620:
bch2_btree_bset_insert_key 于 /home/bhzhu/source/bcachfs-github/bcachefs/fs/bcachefs/btree_trans_commit.c:100 (discriminator 3)
❰bhzhu❙~/source/bcachfs-github/bcachefs(git:master)❱✔≻ export LC_ALL=C
?bhzhu?~/source/bcachfs-github/bcachefs(git:master)??? scripts/faddr2line  fs/bcachefs/bcachefs.ko  bch2_btree_bset_insert_key+0x406/0x620
bch2_btree_bset_insert_key+0x406/0x620:
bch2_btree_bset_insert_key at /home/bhzhu/source/bcachfs-github/bcachefs/fs/bcachefs/btree_trans_commit.c:100 (discriminator 3)
?bhzhu?~/source/bcachfs-github/bcachefs(git:master)??? scripts/faddr2line  fs/bcachefs/bcachefs.ko  bch2_btree_bset_insert_key+0x406/0x620
bch2_btree_bset_insert_key+0x406/0x620:
bch2_btree_bset_insert_key at /home/bhzhu/source/bcachfs-github/bcachefs/fs/bcachefs/btree_trans_commit.c:100 (discriminator 3)
?bhzhu?~/source/bcachfs-github/bcachefs(git:master)??? scripts/faddr2line  fs/bcachefs/bcachefs.ko  bch2_btree_bset_insert_key+0x406/0x620
bch2_btree_bset_insert_key+0x406/0x620:
bch2_btree_bset_insert_key at /home/bhzhu/source/bcachfs-github/bcachefs/fs/bcachefs/btree_trans_commit.c:100 (discriminator 3)
?bhzhu?~/source/bcachfs-github/bcachefs(git:master)??? scripts/faddr2line  fs/bcachefs/bcachefs.ko  bch2_btree_bset_insert_key+0x406/0x620
bch2_btree_bset_insert_key+0x406/0x620:
bch2_btree_bset_insert_key at /home/bhzhu/source/bcachfs-github/bcachefs/fs/bcachefs/btree_trans_commit.c:100 (discriminator 3)
?bhzhu?~/source/bcachfs-github/bcachefs(git:master)??? scripts/faddr2line  fs/bcachefs/bcachefs.ko  bch2_btree_bset_insert_key+0x406/0x620
bch2_btree_bset_insert_key+0x406/0x620:
bch2_btree_bset_insert_key at /home/bhzhu/source/bcachfs-github/bcachefs/fs/bcachefs/btree_trans_commit.c:100 (discriminator 3)
?bhzhu?~/source/bcachfs-github/bcachefs(git:master)??? scripts/faddr2line  fs/bcachefs/bcachefs.ko  bch2_btree_path_verify+0x20/0xd0
bch2_btree_path_verify+0x20/0xd0:
bch2_btree_path_verify at /home/bhzhu/source/bcachfs-github/bcachefs/fs/bcachefs/btree_iter.c:226 (discriminator 1)
?bhzhu?~/source/bcachfs-github/bcachefs(git:master)??? scripts/faddr2line  fs/bcachefs/bcachefs.ko  __bch2_btree_path_set_pos+0x1c9/0x4f0
__bch2_btree_path_set_pos+0x1c9/0x4f0:
__bch2_btree_path_set_pos at /home/bhzhu/source/bcachfs-github/bcachefs/fs/bcachefs/btree_iter.c:1265
?bhzhu?~/source/bcachfs-github/bcachefs(git:master)??? scripts/faddr2line  fs/bcachefs/bcachefs.ko  __bch2_btree_write_buffer_flush+0x355/0xbc0
__bch2_btree_write_buffer_flush+0x355/0xbc0:
bch2_trans_iter_init at /home/bhzhu/source/bcachfs-github/bcachefs/fs/bcachefs/btree_iter.h:482
(inlined by) __bch2_btree_write_buffer_flush at /home/bhzhu/source/bcachfs-github/bcachefs/fs/bcachefs/btree_write_buffer.c:198
?bhzhu?~/source/bcachfs-github/bcachefs(git:master)??? scripts/faddr2line  fs/bcachefs/bcachefs.ko bch2_btree_insert_key_leaf+0x8e/0x280
bch2_btree_insert_key_leaf+0x8e/0x280:
bch2_btree_insert_key_leaf at /home/bhzhu/source/bcachfs-github/bcachefs/fs/bcachefs/btree_trans_commit.c:234 (discriminator 3)
?bhzhu?~/source/bcachfs-github/bcachefs(git:master)??? scripts/faddr2line  __bch2_btree_write_buffer_flush+0x55f/0xbc0
usage: faddr2line [--list] <object file> <func+offset> <func+offset>...
?bhzhu?~/source/bcachfs-github/bcachefs(git:master)??? scripts/faddr2line  fs/bcachefs/bcachefs.ko __bch2_btree_write_buffer_flush+0x55f/0xbc0
__bch2_btree_write_buffer_flush+0x55f/0xbc0:
bch2_btree_write_buffer_flush_one at /home/bhzhu/source/bcachfs-github/bcachefs/fs/bcachefs/btree_write_buffer.c:66
(inlined by) __bch2_btree_write_buffer_flush at /home/bhzhu/source/bcachfs-github/bcachefs/fs/bcachefs/btree_write_buffer.c:206
?bhzhu?~/source/bcachfs-github/bcachefs(git:master)??? scripts/faddr2line  fs/bcachefs/bcachefs.ko     __bch2_trans_commit+0x678/0x1d80
__bch2_trans_commit+0x678/0x1d80:
__bch2_trans_commit at /home/bhzhu/source/bcachfs-github/bcachefs/fs/bcachefs/btree_trans_commit.c:1054
?bhzhu?~/source/bcachfs-github/bcachefs(git:master)??? scripts/faddr2line  fs/bcachefs/bcachefs.ko     bch2_extent_update+0x171/0x230
bch2_extent_update+0x171/0x230:
bch2_extent_update at /home/bhzhu/source/bcachfs-github/bcachefs/fs/bcachefs/io_write.c:310 (discriminator 1)
?bhzhu?~/source/bcachfs-github/bcachefs(git:master)??? scripts/faddr2line  fs/bcachefs/bcachefs.ko    bch2_write_index_default+0x2db/0x430
bch2_write_index_default+0x2db/0x430:
bch2_write_index_default at /home/bhzhu/source/bcachfs-github/bcachefs/fs/bcachefs/io_write.c:361
?bhzhu?~/source/bcachfs-github/bcachefs(git:master)??? scripts/faddr2line  fs/bcachefs/bcachefs.ko    bch2_write_index_default+0x10f/0x430
bch2_write_index_default+0x10f/0x430:
bkey_start_offset at /home/bhzhu/source/bcachfs-github/bcachefs/fs/bcachefs/bkey.h:348
(inlined by) bkey_start_pos at /home/bhzhu/source/bcachfs-github/bcachefs/fs/bcachefs/bkey.h:355
(inlined by) bch2_write_index_default at /home/bhzhu/source/bcachfs-github/bcachefs/fs/bcachefs/io_write.c:350
?bhzhu?~/source/bcachfs-github/bcachefs(git:master)??? scripts/faddr2line  fs/bcachefs/bcachefs.ko    __bch2_write_index+0x191/0x260
__bch2_write_index+0x191/0x260:
__bch2_write_index at /home/bhzhu/source/bcachfs-github/bcachefs/fs/bcachefs/io_write.c:514
?bhzhu?~/source/bcachfs-github/bcachefs(git:master)??? scripts/faddr2line  fs/bcachefs/bcachefs.ko    __bch2_write_index+0x191/0x260
__bch2_write_index+0x191/0x260:
__bch2_write_index at /home/bhzhu/source/bcachfs-github/bcachefs/fs/bcachefs/io_write.c:514
?bhzhu?~/source/bcachfs-github/bcachefs(git:master)??? scripts/faddr2line  fs/bcachefs/bcachefs.ko   bch2_write_point_do_index_updates+0x9d/0x140
bch2_write_point_do_index_updates+0x9d/0x140:
bch2_write_point_do_index_updates at /home/bhzhu/source/bcachfs-github/bcachefs/fs/bcachefs/io_write.c:625
?bhzhu?~/source/bcachfs-github/bcachefs(git:master)???
koverstreet commented 8 months ago

This should be fixed now - squashed the fix into the "bcachefs: rebalance_work btree is not a snapshots btree" patch.