truongsinh
commented
2 months ago This time, it's kernel task of rebalance:
root@qemuvm:~# [ 861.228704] INFO: task bch-rebalance/1:837 blocked for more than 122 seconds.
[ 861.236360] Not tainted 6.11.0-061100rc7daily20240915-generic #202409150203
[ 861.250846] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 987.179736] Allocator stuck? Waited for 30 seconds
[ 987.179752] Allocator debug:
[ 987.179757] capacity8344943698
[ 987.179761] reserved 725647278
[ 987.179766] hidden 16231424
[ 987.179770] btree 55331840
[ 987.179774] data 7198854472
[ 987.179779] cached 0
[ 987.179785] reserved 0
[ 987.179788] online_reserved 0
[ 987.179792] nr_inodes 0
[ 987.179796]
[ 987.179800] freelist_wait waiting
[ 987.179805] open buckets allocated4
[ 987.179809] open buckets total 1024
[ 987.179814] open_buckets_wait empty
[ 987.179818] open_buckets_btree 2
[ 987.179822] open_buckets_user 1
[ 987.179826] btree reserve cache 1
[ 987.179830]
[ 987.179834] Dev 0:
[ 987.179838] buckets sectors fragmented
[ 987.179843] free 67509 0 0
[ 987.179925] sb 13 6152 504
[ 987.179931] journal 8192 4194304 0
[ 987.179936] btree 0 0 0
[ 987.179941] user 4240318 2171042816 0
[ 987.179945] cached 0 0 0
[ 987.179950] parity 0 0 0
[ 987.179954] stripe 0 0 0
[ 987.179959] need_gc_gens 0 0 0
[ 987.179964] need_discard 0 0 0
[ 987.179968] unstriped 0 0 0
[ 987.179973] capacity 4316032
[ 987.179977]
[ 987.179981] reserves:
[ 987.179984] stripe 134932
[ 987.179990] normal 67494
[ 987.179994] copygc 56
[ 987.179999] btree 28
[ 987.180002] btree_copygc 0
[ 987.180007] reclaim 0
[ 987.180011] interior_updates 0
[ 987.180014]
[ 987.180018] open buckets 0
[ 987.180023] buckets to invalidate 0
[ 987.180026]
[ 987.180030] Dev 1:
[ 987.180034] buckets sectors fragmented
[ 987.180039] free 15179 0 0
[ 987.180044] sb 13 6152 504
[ 987.180048] journal 7631 3907072 0
[ 987.180053] btree 26798 13720576 0
[ 987.180058] user 927167 474709504 0
[ 987.180062] cached 0 0 0
[ 987.180067] parity 0 0 0
[ 987.180071] stripe 0 0 0
[ 987.180076] need_gc_gens 0 0 0
[ 987.180081] need_discard 0 0 0
[ 987.180085] unstriped 0 0 0
[ 987.180090] capacity 976788
[ 987.180094]
[ 987.180097] reserves:
[ 987.180101] stripe 30580
[ 987.180105] normal 15318
[ 987.180109] copygc 56
[ 987.180114] btree 28
[ 987.180118] btree_copygc 0
[ 987.180121] reclaim 0
[ 987.180126] interior_updates 0
[ 987.180129]
[ 987.180133] open buckets 1
[ 987.180137] buckets to invalidate 0
[ 987.180141]
[ 987.180144] Dev 2:
[ 987.180148] buckets sectors fragmented
[ 987.180153] free 15190 0 0
[ 987.180158] sb 13 6152 504
[ 987.180162] journal 7631 3907072 0
[ 987.180167] btree 27455 14056960 0
[ 987.180172] user 926499 474367304 184
[ 987.180176] cached 0 0 0
[ 987.180181] parity 0 0 0
[ 987.180185] stripe 0 0 0
[ 987.180190] need_gc_gens 0 0 0
[ 987.180194] need_discard 0 0 0
[ 987.180199] unstriped 0 0 0
[ 987.180203] capacity 976788
[ 987.180207]
[ 987.180211] reserves:
[ 987.180215] stripe 30580
[ 987.180219] normal 15318
[ 987.180223] copygc 56
[ 987.180227] btree 28
[ 987.180231] btree_copygc 0
[ 987.180235] reclaim 0
[ 987.180239] interior_updates 0
[ 987.180242]
[ 987.180246] open buckets 2
[ 987.180250] buckets to invalidate 0
[ 987.180254]
[ 987.180257] Dev 3:
[ 987.180261] buckets sectors fragmented
[ 987.180266] free 119199 0 0
[ 987.180270] sb 13 6152 504
[ 987.180275] journal 8192 4194304 0
[ 987.180280] btree 53817 27554304 0
[ 987.180284] user 7449691 3814241792 0
[ 987.180289] cached 0 0 0
[ 987.180294] parity 0 0 0
[ 987.180298] stripe 0 0 0
[ 987.180303] need_gc_gens 0 0 0
[ 987.180307] need_discard 0 0 0
[ 987.180312] unstriped 0 0 0
[ 987.180316] capacity 7630912
[ 987.180320]
[ 987.180324] reserves:
[ 987.180328] stripe 238522
[ 987.180332] normal 119289
[ 987.180336] copygc 56
[ 987.180340] btree 28
[ 987.180344] btree_copygc 0
[ 987.180347] reclaim 0
[ 987.180352] interior_updates 0
[ 987.180355]
[ 987.180359] open buckets 0
[ 987.180363] buckets to invalidate 0
[ 987.180367]
[ 987.180370] Dev 4:
[ 987.180374] buckets sectors fragmented
[ 987.180379] free 18446744073709294247 0 0
[ 987.180384] sb 1 1024 0
[ 987.180388] journal 1 1024 0
[ 987.180393] btree 0 0 0
[ 987.180397] user 258294 264493056 0
[ 987.180402] cached 0 0 0
[ 987.180406] parity 0 0 0
[ 987.180411] stripe 0 0 0
[ 987.180416] need_gc_gens 0 0 0
[ 987.180420] need_discard 0 0 0
[ 987.180425] unstriped 0 0 0
[ 987.180429] capacity 1907739
[ 987.180433]
[ 987.180437] reserves:
[ 987.180441] stripe 59644
[ 987.180445] normal 29836
[ 987.180449] copygc 28
[ 987.180453] btree 14
[ 987.180457] btree_copygc 0
[ 987.180461] reclaim 0
[ 987.180465] interior_updates 0
[ 987.180469]
[ 987.180472] open buckets 0
[ 987.180476] buckets to invalidate 0
[ 987.180480]
[ 987.180483] Copygc debug:
[ 987.180487] running: 0
[ 987.180491] copygc_wait:0
[ 987.180495] copygc_wait_at:0
[ 987.180499] Currently waiting for:0 B
[ 987.180503] Currently waiting since:5.60 TiB
[ 987.180507] Currently calculated wait:0 B
[ 987.180511]
[ 987.180515] Journal debug:
[ 987.180519] flags: replay_done,running,may_skip_flush
[ 987.180524] dirty journal entries: 0/32768
[ 987.180528] seq: 1031382
[ 987.180532] seq_ondisk: 1031382
[ 987.180536] last_seq: 1031383
[ 987.180540] last_seq_ondisk: 1031382
[ 987.180544] flushed_seq_ondisk: 1031382
[ 987.180548] watermark: stripe
[ 987.180553] each entry reserved: 321
[ 987.180557] nr flush writes: 15
[ 987.180561] nr noflush writes: 0
[ 987.180565] average write size: 6.69 KiB
[ 987.180569] nr direct reclaim: 0
[ 987.180573] nr background reclaim: 145
[ 987.180577] reclaim kicked: 0
[ 987.180581] reclaim runs in: 0 ms
[ 987.180585] blocked: 0
[ 987.180589] current entry sectors: 512
[ 987.180593] current entry error: ok
[ 987.180597] current entry: closed
[ 987.180602] unwritten entries:
[ 987.180606] last buf closed
[ 987.180609] space:
[ 987.180613] discarded 512:4193792
[ 987.180618] clean ondisk 512:4193792
[ 987.180622] clean 512:4193792
[ 987.180626] total 512:4194304
[ 987.180630] dev 0:
[ 987.180634] nr 8192
[ 987.180638] bucket size 512
[ 987.180642] available 8190:512
[ 987.180646] discard_idx 0
[ 987.180651] dirty_ondisk 0 (seq 0)
[ 987.180655] dirty_idx 0 (seq 0)
[ 987.180659] cur_idx 0 (seq 0)
[ 987.180663] dev 1:
[ 987.180667] nr 7631
[ 987.180671] bucket size 512
[ 987.180675] available 7629:336
[ 987.180679] discard_idx 0
[ 987.180683] dirty_ondisk 0 (seq 1031381)
[ 987.180688] dirty_idx 0 (seq 1031381)
[ 987.180693] cur_idx 0 (seq 1031381)
[ 987.180697] dev 2:
[ 987.180700] nr 7631
[ 987.180704] bucket size 512
[ 987.180709] available 7629:48
[ 987.180713] discard_idx 6636
[ 987.180717] dirty_ondisk 6636 (seq 1031382)
[ 987.180721] dirty_idx 6636 (seq 1031382)
[ 987.180726] cur_idx 6636 (seq 1031382)
[ 987.180730] dev 3:
[ 987.180734] nr 8192
[ 987.180738] bucket size 512
[ 987.180742] available 8190:96
[ 987.180746] discard_idx 1574
[ 987.180750] dirty_ondisk 1574 (seq 1031382)
[ 987.180755] dirty_idx 1574 (seq 1031382)
[ 987.180759] cur_idx 1574 (seq 1031382)
[ 987.180763] dev 4:
[ 987.180767] nr 8192
[ 987.180771] bucket size 1024
[ 987.180775] available 8190:1024
[ 987.180779] discard_idx 0
[ 987.180783] dirty_ondisk 0 (seq 0)
[ 987.180787] dirty_idx 0 (seq 0)
[ 987.180791] cur_idx 0 (seq 0)
[ 987.180795] 
koverstreet
gkmcd
I'm running
6.11.0-061100rc7daily20240915-generic, is this problem the lock contention that was rejected during 6.11 RC window?