search

koverstreet / bcachefs

Other
633 stars 69 forks source link

bcachefs data job migrate fails (`non free bucket in freespace btree` followed by OOM kill) #688

Closed cyclingwithelephants closed 2 weeks ago

cyclingwithelephants commented 3 weeks ago

summary

After replacing an ssd with a new one, my machine cannot run bcachefs data job migrate succesfully. Happy to provide any more info I can

update: it looks like my disks have now failed during either this or a subsequent rebuild, so I'm not sure if the original issue is still valid

hardware + kernel

3x8TB HDD /dev/sd{a b c} 1x256GB SSD /dev/sdd 1x4TB SSD /dev/sde

machine:

The RAM might simply be the issue, I've been trying to get this machine working in some fashion before throwing in the towel and getting something a little more capable. If this is the problem, I'd appreciate some guidance on minimum specs (at least RAM wise) for general purpose usage

bcachefs show-super

Device:                                     WDC  WDS400T2B0A
External UUID:                              0eee462f-4912-4c9a-8c5c-b988a4bf0f42
Internal UUID:                              7088f925-4cf2-4693-89ff-613dfb4789d1
Magic number:                               c68573f6-66ce-90a9-d96a-60cf803df7ef
Device index:                               4
Label:                                      bcachefs
Version:                                    1.4: member_seq
Version upgrade complete:                   1.4: member_seq
Oldest version on disk:                     1.4: member_seq
Created:                                    Tue Jun  4 13:08:02 2024
Sequence number:                            112
Time of last write:                         Fri Jun 14 21:53:10 2024
Superblock size:                            7.01 KiB/1.00 MiB
Clean:                                      0
Devices:                                    5
Sections:                                   members_v1,replicas_v0,disk_groups,clean,journal_seq_blacklist,journal_v2,counters,members_v2,errors,ext,downgrade
Features:                                   zstd,journal_seq_blacklist_v3,new_siphash,inline_data,new_extent_overwrite,btree_ptr_v2,extents_above_btree_updates,btree_updates_journalled,new_varint,journal_no_flush,alloc_v2,extents_across_btree_nodes
Compat features:                            alloc_info,alloc_metadata,extents_above_btree_updates_done,bformat_overflow_done

Options:
  block_size:                               4.00 KiB
  btree_node_size:                          256 KiB
  errors:                                   continue [ro] panic
  metadata_replicas:                        2
  data_replicas:                            2
  metadata_replicas_required:               1
  data_replicas_required:                   1
  encoded_extent_max:                       64.0 KiB
  metadata_checksum:                        none [crc32c] crc64 xxhash
  data_checksum:                            none [crc32c] crc64 xxhash
  compression:                              none
  background_compression:                   zstd
  str_hash:                                 crc32c crc64 [siphash]
  metadata_target:                          ssd
  foreground_target:                        ssd
  background_target:                        hdd
  promote_target:                           ssd
  erasure_code:                             0
  inodes_32bit:                             1
  shard_inode_numbers:                      1
  inodes_use_key_cache:                     1
  gc_reserve_percent:                       8
  gc_reserve_bytes:                         0 B
  root_reserve_percent:                     0
  wide_macs:                                0
  acl:                                      1
  usrquota:                                 0
  grpquota:                                 0
  prjquota:                                 0
  journal_flush_delay:                      1000
  journal_flush_disabled:                   0
  journal_reclaim_delay:                    100
  journal_transaction_names:                1
  version_upgrade:                          [compatible] incompatible none
  nocow:                                    0

members_v2 (size 696):
Device:                                     0
  Label:                                    1 (1)
  UUID:                                     fa189b36-f545-4b65-900a-109e0b0ecfdd
  Size:                                     7.28 TiB
  read errors:                              0
  write errors:                             0
  checksum errors:                          0
  seqread iops:                             0
  seqwrite iops:                            0
  randread iops:                            0
  randwrite iops:                           0
  Bucket size:                              256 KiB
  First bucket:                             0
  Buckets:                                  30523541
  Last mount:                               Fri Jun 14 20:27:52 2024
  Last superblock write:                    112
  State:                                    rw
  Data allowed:                             journal,btree,user
  Has data:                                 journal,user,cached
  Durability:                               1
  Discard:                                  0
  Freespace initialized:                    1
Device:                                     1
  Label:                                    2 (2)
  UUID:                                     2f2ed1ac-ffd5-44c1-877e-8338f9e48554
  Size:                                     7.28 TiB
  read errors:                              0
  write errors:                             0
  checksum errors:                          0
  seqread iops:                             0
  seqwrite iops:                            0
  randread iops:                            0
  randwrite iops:                           0
  Bucket size:                              256 KiB
  First bucket:                             0
  Buckets:                                  30523541
  Last mount:                               Fri Jun 14 20:27:52 2024
  Last superblock write:                    112
  State:                                    rw
  Data allowed:                             journal,btree,user
  Has data:                                 journal,btree,user,cached
  Durability:                               1
  Discard:                                  0
  Freespace initialized:                    1
Device:                                     2
  Label:                                    3 (3)
  UUID:                                     af012cff-526c-4b4d-82ea-c8f566f17c40
  Size:                                     7.28 TiB
  read errors:                              0
  write errors:                             0
  checksum errors:                          0
  seqread iops:                             0
  seqwrite iops:                            0
  randread iops:                            0
  randwrite iops:                           0
  Bucket size:                              256 KiB
  First bucket:                             0
  Buckets:                                  30523541
  Last mount:                               Fri Jun 14 20:27:52 2024
  Last superblock write:                    112
  State:                                    rw
  Data allowed:                             journal,btree,user
  Has data:                                 journal,btree,user,cached
  Durability:                               1
  Discard:                                  0
  Freespace initialized:                    1
Device:                                     3
  Label:                                    1 (5)
  UUID:                                     3f8cee15-6a1f-4d30-ab2d-42eb67e832c3
  Size:                                     238 GiB
  read errors:                              0
  write errors:                             0
  checksum errors:                          0
  seqread iops:                             0
  seqwrite iops:                            0
  randread iops:                            0
  randwrite iops:                           0
  Bucket size:                              256 KiB
  First bucket:                             0
  Buckets:                                  976793
  Last mount:                               Fri Jun 14 20:27:52 2024
  Last superblock write:                    112
  State:                                    rw
  Data allowed:                             journal,btree,user
  Has data:                                 journal,btree,user,cached
  Durability:                               1
  Discard:                                  1
  Freespace initialized:                    1
Device:                                     4
  Label:                                    2 (6)
  UUID:                                     c1ec3a7a-5c4f-431b-bd6f-3e751c884b48
  Size:                                     3.64 TiB
  read errors:                              0
  write errors:                             0
  checksum errors:                          0
  seqread iops:                             0
  seqwrite iops:                            0
  randread iops:                            0
  randwrite iops:                           0
  Bucket size:                              1.00 MiB
  First bucket:                             0
  Buckets:                                  3815447
  Last mount:                               Fri Jun 14 20:27:52 2024
  Last superblock write:                    112
  State:                                    rw
  Data allowed:                             journal,btree,user
  Has data:                                 journal,btree,user,cached
  Durability:                               1
  Discard:                                  1
  Freespace initialized:                    1

errors (size 8):

bcachefs fs usage -h

Filesystem: 0eee462f-4912-4c9a-8c5c-b988a4bf0f42
Size:                       23.6 TiB
Used:                       6.02 TiB
Online reserved:                 0 B

Data type       Required/total  Durability    Devices
btree:          1/2             2             [sdd sde]           68.8 GiB
btree:          1/2             2             [sdb sdc]           19.0 MiB
user:           1/2             2             [sdb sde]           7.02 GiB
user:           1/2             2             [sda sdb]            326 GiB
user:           1/2             2             [sdb sdc]           5.29 TiB
user:           1/2             2             [sdc sde]           7.02 GiB
user:           1/2             2             [sda sdc]            324 GiB
user:           1/2             2             [sdb sdd]           30.8 MiB
user:           1/2             2             [sdc sdd]           32.2 MiB
cached:         1/1             1             [sdb]                102 GiB
cached:         1/1             1             [sdd]                191 GiB
cached:         1/1             1             [sda]                387 GiB
cached:         1/1             1             [sdc]                102 GiB
cached:         1/1             1             [sde]               2.57 TiB

hdd.1 (device 0):                sda              rw
                                data         buckets    fragmented
  free:                     6.36 TiB        26657257
  sb:                       3.00 MiB              13       252 KiB
  journal:                  2.00 GiB            8192
  btree:                         0 B               0
  user:                      325 GiB         1352920      5.09 GiB
  cached:                    387 GiB         2505159
  parity:                        0 B               0
  stripe:                        0 B               0
  need_gc_gens:                  0 B               0
  need_discard:                  0 B               0
  capacity:                 7.28 TiB        30523541

hdd.2 (device 1):                sdb              rw
                                data         buckets    fragmented
  free:                     4.07 TiB        17053235
  sb:                       3.00 MiB              13       252 KiB
  journal:                  2.00 GiB            8192
  btree:                    9.50 MiB              38
  user:                     2.81 TiB        11928410      38.4 GiB
  cached:                    102 GiB         1533653
  parity:                        0 B               0
  stripe:                        0 B               0
  need_gc_gens:                  0 B               0
  need_discard:                  0 B               0
  capacity:                 7.28 TiB        30523541

hdd.3 (device 2):                sdc              rw
                                data         buckets    fragmented
  free:                     4.06 TiB        17039707
  sb:                       3.00 MiB              13       252 KiB
  journal:                  2.00 GiB            8192
  btree:                    9.50 MiB              38
  user:                     2.81 TiB        11924368      38.1 GiB
  cached:                    102 GiB         1551223
  parity:                        0 B               0
  stripe:                        0 B               0
  need_gc_gens:                  0 B               0
  need_discard:                  0 B               0
  capacity:                 7.28 TiB        30523541

ssd.1 (device 3):                sdd              rw
                                data         buckets    fragmented
  free:                     9.33 GiB           38211
  sb:                       3.00 MiB              13       252 KiB
  journal:                  1.86 GiB            7631
  btree:                    34.4 GiB          140818
  user:                     31.5 MiB             126
  cached:                    191 GiB          789994
  parity:                        0 B               0
  stripe:                        0 B               0
  need_gc_gens:                  0 B               0
  need_discard:                  0 B               0
  capacity:                  238 GiB          976793

ssd.2 (device 4):                sde              rw
                                data         buckets    fragmented
  free:                      914 GiB          935800
  sb:                       3.00 MiB               4      1020 KiB
  journal:                  8.00 GiB            8192
  btree:                    34.4 GiB          103732      66.9 GiB
  user:                     7.02 GiB           10464      3.19 GiB
  cached:                   2.57 TiB         2757255
  parity:                        0 B               0
  stripe:                        0 B               0
  need_gc_gens:                  0 B               0
  need_discard:                  0 B               0
  capacity:                 3.64 TiB         3815447

dmesg


[ 5139.465551] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.465572] transaction updates for __bch2_write journal seq 0

[ 5139.466084] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.466105] transaction updates for __bch2_write journal seq 0

[ 5139.466712] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.466733] transaction updates for __bch2_write journal seq 0

[ 5139.467403] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.467427] transaction updates for __bch2_write journal seq 0

[ 5139.467864] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.467885] transaction updates for __bch2_write journal seq 0

[ 5139.468791] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.468820] transaction updates for __bch2_write journal seq 0

[ 5139.469399] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.469423] transaction updates for __bch2_write journal seq 0

[ 5139.470131] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.470155] transaction updates for __bch2_write journal seq 0

[ 5139.470603] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.470623] transaction updates for __bch2_write journal seq 0

[ 5139.471364] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.471384] transaction updates for __bch2_write journal seq 0

[ 5139.471942] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.471964] transaction updates for __bch2_write journal seq 0

[ 5139.472624] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.472649] transaction updates for __bch2_write journal seq 0

[ 5139.473248] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.473271] transaction updates for __bch2_write journal seq 0

[ 5139.474179] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.474207] transaction updates for __bch2_write journal seq 0

[ 5139.474619] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.474639] transaction updates for __bch2_write journal seq 0

[ 5139.477032] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.477065] transaction updates for __bch2_write journal seq 0

[ 5139.477649] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.477675] transaction updates for __bch2_write journal seq 0

[ 5139.478522] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.478548] transaction updates for __bch2_write journal seq 0

[ 5139.478965] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.478987] transaction updates for __bch2_write journal seq 0

[ 5139.479801] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.479821] transaction updates for __bch2_write journal seq 0

[ 5139.480214] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.480236] transaction updates for __bch2_write journal seq 0

[ 5139.483386] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.483420] transaction updates for __bch2_write journal seq 0

[ 5139.483793] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.483815] transaction updates for __bch2_write journal seq 0

[ 5139.484322] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.484380] transaction updates for __bch2_write journal seq 0

[ 5139.484971] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.484994] transaction updates for __bch2_write journal seq 0

[ 5139.485798] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.485821] transaction updates for __bch2_write journal seq 0

[ 5139.486206] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.486228] transaction updates for __bch2_write journal seq 0

[ 5139.487025] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.487048] transaction updates for __bch2_write journal seq 0

[ 5139.487494] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.487515] transaction updates for __bch2_write journal seq 0

[ 5139.490983] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.491017] transaction updates for __bch2_write journal seq 0

[ 5139.491472] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.491495] transaction updates for __bch2_write journal seq 0

[ 5139.494812] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.494848] transaction updates for __bch2_write journal seq 0

[ 5139.495357] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.495383] transaction updates for __bch2_write journal seq 0

[ 5139.496128] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.496152] transaction updates for __bch2_write journal seq 0

[ 5139.496699] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.496720] transaction updates for __bch2_write journal seq 0

[ 5139.497458] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.497486] transaction updates for __bch2_write journal seq 0

[ 5139.497946] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.497972] transaction updates for __bch2_write journal seq 0

[ 5139.498805] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.498828] transaction updates for __bch2_write journal seq 0

[ 5139.499239] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.499260] transaction updates for __bch2_write journal seq 0

[ 5139.500090] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.500112] transaction updates for __bch2_write journal seq 0

[ 5139.500523] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.500547] transaction updates for __bch2_write journal seq 0

[ 5139.501344] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.501372] transaction updates for __bch2_write journal seq 0

[ 5139.503813] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.503844] transaction updates for __bch2_write journal seq 0

[ 5139.507449] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.507487] transaction updates for __bch2_write journal seq 0

[ 5139.509107] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): bch2_btree_update_start(): error EIO
[ 5139.509132] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073744092 offset 32555008): write error while doing btree update: EIO
[ 5139.509246] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): bch2_btree_update_start(): error EIO
[ 5139.509259] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073744092 offset 32624640): write error while doing btree update: EIO
[ 5139.509369] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): bch2_btree_update_start(): error EIO
[ 5139.509380] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073744092 offset 33349632): write error while doing btree update: EIO
[ 5139.509482] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): bch2_btree_update_start(): error EIO
[ 5139.509495] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073744092 offset 33411072): write error while doing btree update: EIO
[ 5139.509555] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.509575] transaction updates for __bch2_write journal seq 0

[ 5139.510738] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): bch2_btree_update_start(): error EIO
[ 5139.510781] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073744092 offset 34521088): write error while doing btree update: EIO
[ 5139.511329] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): bch2_btree_update_start(): error EIO
[ 5139.511345] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073744092 offset 34537472): write error while doing btree update: EIO
[ 5139.511664] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): bch2_btree_update_start(): error EIO
[ 5139.511679] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073744092 offset 34607104): write error while doing btree update: EIO
[ 5139.511792] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): bch2_btree_update_start(): error EIO
[ 5139.511805] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073744092 offset 35307520): write error while doing btree update: EIO
[ 5139.512010] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): bch2_btree_update_start(): error EIO
[ 5139.512024] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073744092 offset 35373056): write error while doing btree update: EIO
[ 5139.512430] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): bch2_btree_update_start(): error EIO
[ 5139.512448] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073744092 offset 35393536): write error while doing btree update: EIO
[ 5139.685170] ------------[ cut here ]------------
[ 5139.685184] btree trans held srcu lock (delaying memory reclaim) for 42 seconds
[ 5139.685257] WARNING: CPU: 4 PID: 10187 at fs/bcachefs/btree_iter.c:2825 bch2_trans_put+0x24c/0x254 [bcachefs]
[ 5139.685344] Modules linked in: xt_mark xt_nat xt_tcpudp veth xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xfrm_user xfrm_algo xt_addrtype nft_compat br_netfilter bridge nf_tables rfkill bcachefs panfrost gpio_charger gpu_sched lz4hc leds_pwm rockchip_vdec(C) hantro_vpu pwm_fan drm_shmem_helper v4l2_vp9 rockchip_rga videobuf2_dma_contig v4l2_h264 v4l2_mem2mem videobuf2_dma_sg lz4 snd_soc_rockchip_i2s snd_soc_hdmi_codec videobuf2_memops videobuf2_v4l2 snd_soc_core videodev snd_compress snd_pcm_dmaengine videobuf2_common rk_crypto snd_pcm rng_core mc snd_timer snd soundcore gpio_beeper cpufreq_dt zram binfmt_misc sg ledtrig_netdev lm75 nfsd auth_rpcgss nfs_acl lockd grace dm_mod sunrpc nfnetlink ip_tables x_tables autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid1 raid0 md_mod cdc_ncm cdc_ether usbnet r8152 realtek fusb302 tcpm typec dwmac_rk stmmac_platform stmmac pcs_xpcs adc_keys
[ 5139.685640] CPU: 4 PID: 10187 Comm: kworker/4:2 Tainted: G        WC         6.8.12-edge-rockchip64 #1
[ 5139.685650] Hardware name: Helios64 (DT)
[ 5139.685655] Workqueue: bcachefs_btree_io btree_node_write_work [bcachefs]
[ 5139.685720] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 5139.685728] pc : bch2_trans_put+0x24c/0x254 [bcachefs]
[ 5139.685784] lr : bch2_trans_put+0x24c/0x254 [bcachefs]
[ 5139.685839] sp : ffff800083e83d10
[ 5139.685844] x29: ffff800083e83d10 x28: ffff800081ad79c0 x27: ffff800081ad7000
[ 5139.685858] x26: ffff00008fffeb80 x25: ffff00008fffebc0 x24: 0000000000000000
[ 5139.685871] x23: ffff80007a4766a4 x22: ffff00004036d340 x21: ffff000007a80000
[ 5139.685883] x20: ffff000007a80000 x19: ffff000077788000 x18: ffffffffffffffff
[ 5139.685896] x17: 0000000000000000 x16: 0000000000000028 x15: ffff800081d3ee5c
[ 5139.685908] x14: ffffffffffffffff x13: 00000000000017d5 x12: 00000000ffffffea
[ 5139.685921] x11: 0000000000004860 x10: 0000000000004828 x9 : ffff800081b56998
[ 5139.685934] x8 : 000000000000bc10 x7 : c0000000fffff7d6 x6 : 000000000002b190
[ 5139.685947] x5 : ffff0000f7769dc8 x4 : 0000000000000000 x3 : 0000000000000027
[ 5139.685959] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff000057f5a300
[ 5139.685972] Call trace:
[ 5139.685977]  bch2_trans_put+0x24c/0x254 [bcachefs]
[ 5139.686035]  btree_node_write_work+0x200/0x480 [bcachefs]
[ 5139.686092]  process_one_work+0x160/0x3a8
[ 5139.686118]  worker_thread+0x32c/0x438
[ 5139.686127]  kthread+0x114/0x118
[ 5139.686134]  ret_from_fork+0x10/0x20
[ 5139.686144] ---[ end trace 0000000000000000 ]---
[ 5139.686646] ------------[ cut here ]------------
[ 5139.686672] btree trans held srcu lock (delaying memory reclaim) for 42 seconds
[ 5139.686809] WARNING: CPU: 0 PID: 1003 at fs/bcachefs/btree_iter.c:2825 bch2_trans_put+0x24c/0x254 [bcachefs]
[ 5139.686974] Modules linked in: xt_mark xt_nat xt_tcpudp veth xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xfrm_user xfrm_algo xt_addrtype nft_compat br_netfilter bridge nf_tables rfkill bcachefs panfrost gpio_charger gpu_sched lz4hc leds_pwm rockchip_vdec(C) hantro_vpu pwm_fan drm_shmem_helper v4l2_vp9 rockchip_rga videobuf2_dma_contig v4l2_h264 v4l2_mem2mem videobuf2_dma_sg lz4 snd_soc_rockchip_i2s snd_soc_hdmi_codec videobuf2_memops videobuf2_v4l2 snd_soc_core videodev snd_compress snd_pcm_dmaengine videobuf2_common rk_crypto snd_pcm rng_core mc snd_timer snd soundcore gpio_beeper cpufreq_dt zram binfmt_misc sg ledtrig_netdev lm75 nfsd auth_rpcgss nfs_acl lockd grace dm_mod sunrpc nfnetlink ip_tables x_tables autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid1 raid0 md_mod cdc_ncm cdc_ether usbnet r8152 realtek fusb302 tcpm typec dwmac_rk stmmac_platform stmmac pcs_xpcs
[ 5139.687318] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.687329]  adc_keys
[ 5139.687342] transaction updates for __bch2_write journal seq 0
[ 5139.687342] CPU: 0 PID: 1003 Comm: bch-reclaim/0ee Tainted: G        WC         6.8.12-edge-rockchip64 #1

[ 5139.687353] Hardware name: Helios64 (DT)
[ 5139.687360] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 5139.687371] pc : bch2_trans_put+0x24c/0x254 [bcachefs]
[ 5139.687467] lr : bch2_trans_put+0x24c/0x254 [bcachefs]
[ 5139.687558] sp : ffff800083ca3c50
[ 5139.687564] x29: ffff800083ca3c50 x28: ffff80007a3a9f3c x27: 0000000000302236
[ 5139.687580] x26: ffff0000635e4c30 x25: ffff0000635e4bb0 x24: 0000000000000001
[ 5139.687596] x23: 0000000000301eae x22: 0000000000000001 x21: 0000000000000000
[ 5139.687611] x20: ffff000007a80000 x19: ffff0000bd4c8000 x18: ffffffffffffffff
[ 5139.687627] x17: 0000000000000000 x16: 0000000000000028 x15: ffff800081d3f95c
[ 5139.687642] x14: ffffffffffffffff x13: 00000000000017f2 x12: 00000000ffffffea
[ 5139.687657] x11: 0000000000004ed0 x10: 0000000000004e98 x9 : ffff800081b56998
[ 5139.687672] x8 : 000000000000bec8 x7 : c0000000fffff7f3 x6 : 000000000002bb88
[ 5139.687687] x5 : ffff0000f76eddc8 x4 : 0000000000000000 x3 : 0000000000000027
[ 5139.687702] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff0000056d9180
[ 5139.687718] Call trace:
[ 5139.687725]  bch2_trans_put+0x24c/0x254 [bcachefs]
[ 5139.687814]  bch2_btree_key_cache_journal_flush+0xf8/0x208 [bcachefs]
[ 5139.687902]  journal_flush_pins.constprop.0+0x180/0x298 [bcachefs]
[ 5139.687992]  __bch2_journal_reclaim+0x1ec/0x408 [bcachefs]
[ 5139.688086]  bch2_journal_reclaim_thread+0x7c/0x170 [bcachefs]
[ 5139.688178]  kthread+0x114/0x118
[ 5139.688193]  ret_from_fork+0x10/0x20
[ 5139.688206] ---[ end trace 0000000000000000 ]---
[ 5139.697661] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42): non free bucket in freespace btree
                 freespace key u64s 5 type set 0:245104:0 len 1 ver 0
                 u64s 12 type alloc_v4 0:245103:0 len 0 ver 0:
                 gen 2 oldest_gen 0 data_type user
                 journal_seq       3158480
                 need_discard      1
                 need_inc_gen      1
                 dirty_sectors     456
                 cached_sectors    0
                 stripe            0
                 stripe_redundancy 0
                 io_time[READ]     23467768264
                 io_time[WRITE]    28609261512
                 fragmentation     1912602624
                 bp_start          7
[ 5139.697690] transaction updates for __bch2_write journal seq 0

[ 5140.400639] systemd-journald[1039]: Under memory pressure, flushing caches.
cyclingwithelephants commented 3 weeks ago

resuming again produced different dmesg output. It looks like 2 of my disks have failed during this rebuild. I was hoping to confirm this? Interestingly, my install of scrutiny didn't alert me to this

smartctl for /dev/sdc

smartctl 7.4 2023-08-01 r5530 [aarch64-linux-6.8.12-edge-rockchip64] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Ultrastar (He10/12)
Device Model:     WDC WD80EMAZ-00WJTA0
Serial Number:    1EGE740Z
LU WWN Device Id: 5 000cca 27ec60396
Firmware Version: 83.H0A83
User Capacity:    8,001,563,222,016 bytes [8.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database 7.3/5528
ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Fri Jun 14 22:37:56 2024 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x80) Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:        (   93) seconds.
Offline data collection
capabilities:            (0x5b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    No Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:    (   2) minutes.
Extended self-test routine
recommended polling time:    (1069) minutes.
SCT capabilities:          (0x003d) SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0004   127   127   054    Old_age   Offline      -       120
  3 Spin_Up_Time            0x0007   188   188   024    Pre-fail  Always       -       361 (Average 337)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       486
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000a   100   100   067    Old_age   Always       -       0
  8 Seek_Time_Performance   0x0004   128   128   020    Old_age   Offline      -       18
  9 Power_On_Hours          0x0012   097   097   000    Old_age   Always       -       24750
 10 Spin_Retry_Count        0x0012   100   100   060    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       296
 22 Helium_Level            0x0023   100   100   025    Pre-fail  Always       -       100
192 Power-Off_Retract_Count 0x0032   089   089   000    Old_age   Always       -       13476
193 Load_Cycle_Count        0x0012   089   089   000    Old_age   Always       -       13476
194 Temperature_Celsius     0x0002   224   224   000    Old_age   Always       -       29 (Min/Max 13/38)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       3

SMART Error Log Version: 1
ATA Error Count: 3
    CR = Command Register [HEX]
    FR = Features Register [HEX]
    SC = Sector Count Register [HEX]
    SN = Sector Number Register [HEX]
    CL = Cylinder Low Register [HEX]
    CH = Cylinder High Register [HEX]
    DH = Device/Head Register [HEX]
    DC = Device Command Register [HEX]
    ER = Error register [HEX]
    ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 3 occurred at disk power-on lifetime: 24630 hours (1026 days + 6 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 43 00 00 00 00 00  Error: ICRC, ABRT at LBA = 0x00000000 = 0

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 78 20 38 0a 86 40 08      02:09:41.558  READ FPDMA QUEUED
  61 50 70 88 ea f7 40 08      02:09:41.523  WRITE FPDMA QUEUED
  60 80 68 38 1e 86 40 08      02:09:41.521  READ FPDMA QUEUED
  60 40 60 c8 18 86 40 08      02:09:41.520  READ FPDMA QUEUED
  60 80 58 48 18 86 40 08      02:09:41.520  READ FPDMA QUEUED

Error 2 occurred at disk power-on lifetime: 24561 hours (1023 days + 9 hours)
  When the command that caused the error occurred, the device was doing SMART Offline or Self-test.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 43 00 00 00 00 00  Error: ICRC, ABRT at LBA = 0x00000000 = 0

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 80 08 50 59 01 40 08   1d+20:15:38.609  READ FPDMA QUEUED
  60 80 28 80 3a 01 40 08   1d+20:15:38.579  READ FPDMA QUEUED
  60 80 20 00 3a 01 40 08   1d+20:15:38.578  READ FPDMA QUEUED
  60 20 18 80 5a 01 40 08   1d+20:15:38.578  READ FPDMA QUEUED
  60 30 10 d0 59 01 40 08   1d+20:15:38.576  READ FPDMA QUEUED

Error 1 occurred at disk power-on lifetime: 24534 hours (1022 days + 6 hours)
  When the command that caused the error occurred, the device was doing SMART Offline or Self-test.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 43 00 00 00 00 00  Error: ICRC, ABRT at LBA = 0x00000000 = 0

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 80 00 a8 fe a4 40 08      17:13:01.042  READ FPDMA QUEUED
  61 18 20 30 30 ca 40 08      17:13:01.005  WRITE FPDMA QUEUED
  61 10 18 20 30 ca 40 08      17:13:01.004  WRITE FPDMA QUEUED
  61 20 10 00 30 ca 40 08      17:13:01.004  WRITE FPDMA QUEUED
  61 60 08 a0 d1 c3 40 08      17:13:01.004  WRITE FPDMA QUEUED

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     12302         -
# 2  Short offline       Completed without error       00%     12233         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

The above only provides legacy SMART information - try 'smartctl -x' for more

dmesg output

<truncated>
[ 6572.220573] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763141 offset 790491136): __bch2_write(): error: insufficient_devices
[ 6572.220603] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763141 offset 790601728): __bch2_write(): error: insufficient_devices
[ 6572.220617] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763141 offset 790630400): __bch2_write(): error: insufficient_devices
[ 6572.220632] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763141 offset 790695936): __bch2_write(): error: insufficient_devices
[ 6572.220643] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763141 offset 790716416): __bch2_write(): error: insufficient_devices
[ 6572.223383] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763141 offset 790781952): __bch2_write(): error: insufficient_devices
[ 6572.223413] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763141 offset 791220224): __bch2_write(): error: insufficient_devices
[ 6572.223426] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763141 offset 791240704): __bch2_write(): error: insufficient_devices
[ 6572.223435] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763141 offset 791244800): __bch2_write(): error: insufficient_devices
[ 6572.223450] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763141 offset 791310336): __bch2_write(): error: insufficient_devices
[ 6577.228043] __bch2_write: 13794 callbacks suppressed
[ 6577.228054] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763169 offset 10010624): __bch2_write(): error: insufficient_devices
[ 6577.228081] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763169 offset 10022912): __bch2_write(): error: insufficient_devices
[ 6577.228097] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763169 offset 10076160): __bch2_write(): error: insufficient_devices
[ 6577.228110] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763169 offset 10448896): __bch2_write(): error: insufficient_devices
[ 6577.228123] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763169 offset 10485760): __bch2_write(): error: insufficient_devices
[ 6577.228137] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763169 offset 10534912): __bch2_write(): error: insufficient_devices
[ 6577.228147] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763169 offset 10711040): __bch2_write(): error: insufficient_devices
[ 6577.228161] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763169 offset 10756096): __bch2_write(): error: insufficient_devices
[ 6577.228173] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763169 offset 10797056): __bch2_write(): error: insufficient_devices
[ 6577.228184] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763169 offset 10809344): __bch2_write(): error: insufficient_devices
[ 6582.244465] __bch2_write: 15954 callbacks suppressed
[ 6582.244477] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763546 offset 211378176): __bch2_write(): error: insufficient_devices
[ 6582.244590] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763546 offset 211398656): __bch2_write(): error: insufficient_devices
[ 6582.244701] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763546 offset 211423232): __bch2_write(): error: insufficient_devices
[ 6582.244745] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763546 offset 211439616): __bch2_write(): error: insufficient_devices
[ 6582.244759] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763546 offset 212959232): __bch2_write(): error: insufficient_devices
[ 6582.244776] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763546 offset 212996096): __bch2_write(): error: insufficient_devices
[ 6582.244792] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763546 offset 213061632): __bch2_write(): error: insufficient_devices
[ 6582.244826] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763546 offset 213127168): __bch2_write(): error: insufficient_devices
[ 6582.244836] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763546 offset 213131264): __bch2_write(): error: insufficient_devices
[ 6582.244858] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763546 offset 213164032): __bch2_write(): error: insufficient_devices
[ 6587.252207] __bch2_write: 14795 callbacks suppressed
[ 6587.252221] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763550 offset 1540120576): __bch2_write(): error: insufficient_devices
[ 6587.252274] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763550 offset 1540186112): __bch2_write(): error: insufficient_devices
[ 6587.252293] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763550 offset 1540247552): __bch2_write(): error: insufficient_devices
[ 6587.252313] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763550 offset 1540313088): __bch2_write(): error: insufficient_devices
[ 6587.252330] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763550 offset 1540378624): __bch2_write(): error: insufficient_devices
[ 6587.252341] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763550 offset 1540382720): __bch2_write(): error: insufficient_devices
[ 6587.252359] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763550 offset 1540448256): __bch2_write(): error: insufficient_devices
[ 6587.252374] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763550 offset 1540481024): __bch2_write(): error: insufficient_devices
[ 6587.252392] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763550 offset 1540509696): __bch2_write(): error: insufficient_devices
[ 6587.252410] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763550 offset 1540575232): __bch2_write(): error: insufficient_devices
[ 6592.274324] __bch2_write: 13787 callbacks suppressed
[ 6592.274338] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763936 offset 200269824): __bch2_write(): error: insufficient_devices
[ 6592.274404] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763936 offset 200314880): __bch2_write(): error: insufficient_devices
[ 6592.274418] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763936 offset 200486912): __bch2_write(): error: insufficient_devices
[ 6592.274430] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763936 offset 200495104): __bch2_write(): error: insufficient_devices
[ 6592.274440] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763936 offset 200503296): __bch2_write(): error: insufficient_devices
[ 6592.274466] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763936 offset 200568832): __bch2_write(): error: insufficient_devices
[ 6592.274478] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763936 offset 200589312): __bch2_write(): error: insufficient_devices
[ 6592.274492] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763936 offset 200634368): __bch2_write(): error: insufficient_devices
[ 6592.274504] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763936 offset 200654848): __bch2_write(): error: insufficient_devices
[ 6592.274518] bcachefs (0eee462f-4912-4c9a-8c5c-b988a4bf0f42 inum 1073763936 offset 200720384): __bch2_write(): error: insufficient_devices
cyclingwithelephants commented 2 weeks ago

Looks like it was a loose SATA cable 🫠