maartenvanmalland
commented
2 months ago Inspired by https://github.com/koverstreet/bcachefs/issues/672, here some extra info as the effect of this crash is indeed a hanging filesystem as well:
# cat /sys/fs/bcachefs/*/internal/journal_debug
dirty journal entries: 763/32768
seq: 6003
seq_ondisk: 5999
last_seq: 5241
last_seq_ondisk: 5241
flushed_seq_ondisk: 5890
watermark: stripe
each entry reserved: 401
nr flush writes: 284
nr noflush writes: 5699
average write size: 246 KiB
nr direct reclaim: 0
nr background reclaim: 544622
reclaim kicked: 0
reclaim runs in: 12 ms
blocked: 0
current entry sectors: 512
current entry error: ok
current entry: closed
unwritten entries:
seq: 6000
refcount: 0
size: 136 KiB
expires: -135360 jiffies
flags: must_flush separate_flush write_started write allocated
seq: 6001
refcount: 0
size: 255 KiB
expires: -135109 jiffies
flags: noflush separate_flush write_started write allocated write done
seq: 6002
refcount: 0
size: 255 KiB
expires: -135109 jiffies
flags: noflush separate_flush write_started write allocated write done
seq: 6003
refcount: 0
size: 255 KiB
expires: -135109 jiffies
flags: noflush separate_flush write_started write allocated write done
last buf closed
replay done: 1
space:
discarded 512:4193280
clean ondisk 512:4193280
clean 512:4193280
total 512:4194304
dev 0:
nr 1582
bucket size 512
available 818:0
discard_idx 307
dirty_ondisk 307 (seq 5241)
dirty_idx 307 (seq 5241)
cur_idx 1069 (seq 6003)
dev 1:
nr 8192
bucket size 512
available 8190:0
discard_idx 0
dirty_ondisk 0 (seq 0)
dirty_idx 0 (seq 0)
cur_idx 0 (seq 0)
# cat /sys/fs/bcachefs/caa656c1-a03a-4fda-a11d-dd1f0ac8d896/dev-0/alloc_debug
buckets sectors fragmented
free 129877 0 0
sb 13 6152 504
journal 1582 809984 0
btree 106 54272 0
user 5898 3013288 6488
cached 801 410912 0
parity 0 0 0
stripe 0 0 0
need_gc_gens 0 0 0
need_discard 64291 0 0
reserves:
stripe 6386
normal 3221
copygc 56
btree 28
btree_copygc 0
reclaim 0
interior_updates 0
freelist_wait empty
open buckets allocated 107
open buckets this dev 101
open buckets total 1024
open_buckets_wait empty
open_buckets_btree 28
open_buckets_user 78
buckets_to_invalidate 0
btree reserve cache 28`
# cat /sys/kernel/debug/bcachefs/caa656c1-a03a-4fda-a11d-dd1f0ac8d896/btree_transactions
2075159 bch2_moving_ctxt_init
backtrace:
[<0>] kthread+0xde/0x110
[<0>] ret_from_fork+0x31/0x50
[<0>] ret_from_fork_asm+0x1a/0x30
2075160 bch2_moving_ctxt_init
backtrace:
[<0>] bch2_rebalance_thread+0x66/0xb0 [bcachefs]
[<0>] kthread+0xde/0x110
[<0>] ret_from_fork+0x31/0x50
[<0>] ret_from_fork_asm+0x1a/0x30
2177571 bch2_do_discards_work
backtrace:
[<0>] bch2_discard_one_bucket+0x88b/0x8c0 [bcachefs]
[<0>] bch2_do_discards_work+0x1f5/0x3a0 [bcachefs]
[<0>] process_one_work+0x189/0x3b0
[<0>] worker_thread+0x277/0x390
[<0>] kthread+0xde/0x110
[<0>] ret_from_fork+0x31/0x50
[<0>] ret_from_fork_asm+0x1a/0x30
2182976 btree_node_write_work
backtrace:
[<0>] __bch2_trans_commit+0xd18/0x16f0 [bcachefs]
[<0>] __bch2_btree_node_update_key+0x28c/0x570 [bcachefs]
[<0>] bch2_btree_node_update_key+0x35f/0x3a0 [bcachefs]
[<0>] bch2_btree_node_update_key_get_iter+0x1d5/0x330 [bcachefs]
[<0>] btree_node_write_work+0x10a/0x460 [bcachefs]
[<0>] process_one_work+0x189/0x3b0
[<0>] worker_thread+0x277/0x390
[<0>] kthread+0xde/0x110
[<0>] ret_from_fork+0x31/0x50
[<0>] ret_from_fork_asm+0x1a/0x30
3914811 btree_node_write_work
backtrace:
[<0>] __bch2_trans_commit+0xd18/0x16f0 [bcachefs]
[<0>] __bch2_btree_node_update_key+0x28c/0x570 [bcachefs]
[<0>] bch2_btree_node_update_key+0x35f/0x3a0 [bcachefs]
[<0>] bch2_btree_node_update_key_get_iter+0x1d5/0x330 [bcachefs]
[<0>] btree_node_write_work+0x10a/0x460 [bcachefs]
[<0>] process_one_work+0x189/0x3b0
[<0>] worker_thread+0x277/0x390
[<0>] kthread+0xde/0x110
[<0>] ret_from_fork+0x31/0x50
[<0>] ret_from_fork_asm+0x1a/0x30Full dmesg output: http://vpaste.net/FineR
koverstreet
NHOrus
russells-crockpot
On my NAS I have the following setup:
I guess this is one of those setups that you might be thinking, oh dear, what kind of users am I attracting here :). Anyways, this setup performs like a madman, up until the point it crashes/hangs:
This only happens when there is intense writing (and I'm guessing when the data begins to overflow to the hdd target).