search

koverstreet / bcachefs

Other
662 stars 70 forks source link

bcachefs fsck on raid1 does not detect nor fix the errors #606

Closed Mjasnik closed 10 months ago

Mjasnik commented 10 months ago

Version The last commit for kernel 6.5, I can not find the commit anymore, but this was happening before too.

Generic info

bcachefs fs usage
Filesystem: bc9b7455-7f74-47b1-8e78-65a58c3d07db
Size:                    39513699328
Used:                    32986152960
Online reserved:                   0

Data type       Required/total  Devices
btree:          1/2             [sda5 sdb5]                179306496
user:           1/2             [sda5 sdb5]              32464486400

sea1.bch (device 0):            sda5              rw
                                data         buckets    fragmented
  free:                            0           19003
  sb:                        3149824              13        258048
  journal:                 167772160             640
  btree:                    89653248             342
  user:                  16232243200           61922        237568
  cached:                          0               0
  parity:                          0               0
  stripe:                          0               0
  need_gc_gens:                    0               0
  need_discard:                    0               0
  erasure coded:                   0               0
  capacity:              21474836480           81920

sea2.bch (device 1):            sdb5              rw
                                data         buckets    fragmented
  free:                            0           19003
  sb:                        3149824              13        258048
  journal:                 167772160             640
  btree:                    89653248             342
  user:                  16232243200           61922        237568
  cached:                          0               0
  parity:                          0               0
  stripe:                          0               0
  need_gc_gens:                    0               0
  need_discard:                    0               0
  erasure coded:                   0               0
  capacity:              21474836480           81920
bcachefs show-super
External UUID:                              bc9b7455-7f74-47b1-8e78-65a58c3d07db
Internal UUID:                              5ce964f8-d550-434f-8441-5543b29b3fd1
Device index:                               0
Label:                                      BIGDATA2.BCH
Version:                                    1.3: rebalance_work
Version upgrade complete:                   1.3: rebalance_work
Oldest version on disk:                     1.3: rebalance_work
Created:                                    Fri Nov  3 22:52:16 2023

Sequence number:                            41
Superblock size:                            5088
Clean:                                      0
Devices:                                    2
Sections:                                   members_v1,replicas_v0,disk_groups,clean,journal_v2,counters,members_v2,errors
Features:                                   new_siphash,inline_data,new_extent_overwrite,btree_ptr_v2,extents_above_btree_updates,btree_updates_journalled,new_varint,journal_no_flush,alloc_v2,extents_across_btree_nodes
Compat features:                            alloc_info,alloc_metadata,extents_above_btree_updates_done,bformat_overflow_done

Options:
  block_size:                               4.00 KiB
  btree_node_size:                          256 KiB
  errors:                                   continue [ro] panic 
  metadata_replicas:                        2
  data_replicas:                            2
  metadata_replicas_required:               1
  data_replicas_required:                   1
  encoded_extent_max:                       64.0 KiB
  metadata_checksum:                        none [crc32c] crc64 xxhash 
  data_checksum:                            none [crc32c] crc64 xxhash 
  compression:                              none
  background_compression:                   none
  str_hash:                                 crc32c crc64 [siphash] 
  metadata_target:                          none
  foreground_target:                        none
  background_target:                        none
  promote_target:                           none
  erasure_code:                             0
  inodes_32bit:                             1
  shard_inode_numbers:                      1
  inodes_use_key_cache:                     1
  gc_reserve_percent:                       8
  gc_reserve_bytes:                         0 B
  root_reserve_percent:                     0
  wide_macs:                                0
  acl:                                      1
  usrquota:                                 0
  grpquota:                                 0
  prjquota:                                 0
  journal_flush_delay:                      1000
  journal_flush_disabled:                   0
  journal_reclaim_delay:                    100
  journal_transaction_names:                1
  version_upgrade:                          [compatible] incompatible none 
  nocow:                                    0

members_v2 (size 256):
  Device:                                   0
    Label:                                  bch (1)
    UUID:                                   5cfeb555-a2dc-4ad1-ac1a-018fdc6340aa
    Size:                                   20.0 GiB
    read errors:                            0
    write errors:                           0
    checksum errors:                        0
    seqread iops:                           0
    seqwrite iops:                          0
    randread iops:                          0
    randwrite iops:                         0
    Bucket size:                            256 KiB
    First bucket:                           0
    Buckets:                                81920
    Last mount:                             Sat Nov  4 00:35:22 2023

    State:                                  rw
    Data allowed:                           journal,btree,user
    Has data:                               journal,btree,user
    Discard:                                0
    Freespace initialized:                  1
  Device:                                   1
    Label:                                  bch (3)
    UUID:                                   a5bbcd81-035a-40e7-ad38-47ed168e20b7
    Size:                                   20.0 GiB
    read errors:                            0
    write errors:                           0
    checksum errors:                        653
    seqread iops:                           0
    seqwrite iops:                          0
    randread iops:                          0
    randwrite iops:                         0
    Bucket size:                            256 KiB
    First bucket:                           0
    Buckets:                                81920
    Last mount:                             Sat Nov  4 00:35:22 2023

    State:                                  rw
    Data allowed:                           journal,btree,user
    Has data:                               journal,btree,user
    Discard:                                0
    Freespace initialized:                  1

replicas_v0 (size 24):
  btree: 2 [0 1] journal: 2 [0 1] user: 2 [0 1]

The issue The raid1 errors do not get detected by fsck, fsck does not find and fix them, there is no scrub or rebalance, how to fix them? The situation is created artificially, sort of simulating huge bitrot.

sudo bcachefs format --replicas=2 -L BIGDATA2.BCH --label=sea1.bch /dev/disk/by-partlabel/sea1.bch1 --label=sea2.bch /dev/disk/by-partlabel/sea2.bch1
sudo mkdir /mnt/BIGDATA2.BCH
sudo mount -t bcachefs /dev/disk/by-partlabel/sea1.bch1:/dev/disk/by-partlabel/sea2.bch1 /mnt/BIGDATA2.BCH
sudo chown user:user /mnt/BIGDATA2.BCH
time rsync --info=progress2 --fsync -a "/mnt/DATA/WinePrefixes/GOG.Galaxy/drive_c/Program Files (x86)/GOG Galaxy/Games/Cyberpunk 2077/archive/pc/content/basegame_4_appearance.archive" /mnt/BIGDATA2.BCH/
sudo umount /mnt/BIGDATA.BCH
sudo dd if=/dev/urandom of=/dev/disk/by-partlabel/sea2.bch1 bs=1024000 count=10 oflag=direct seek=10240
sudo mount -t bcachefs /dev/disk/by-partlabel/sea1.bch1:/dev/disk/by-partlabel/sea2.bch1 /mnt/BIGDATA2.BCH

Doing this

time md5sum /mnt/BIGDATA2.BCH/basegame_4_appearance.archive

results in errors in kernel log, md5sum is correct, i.e. it reads the file correctly

nov 04 00:36:47 crazy kernel: bcachefs (sdb5 inum 67108864 offset 10256252928): data checksum error: expected 0:6d7053e4 got 0:4899c936 (type crc32c)
...

Fsck does not help

sudo bcachefs fsck -pvy /dev/disk/by-partlabel/sea2.bch1 /dev/disk/by-partlabel/sea1.bch1mounting version 1.3: rebalance_work opts=metadata_replicas=2,data_replicas=2,degraded,verbose,fsck,fix_errors=yes
recovering from clean shutdown, journal seq 1508
starting journal read
journal read done on device 0x55f00d8f5750g, ret 0
journal read done on device 0x55f00d8f5380g, ret 0
journal read done, replaying entries 1508-1508
alloc_read... done
stripes_read... done
snapshots_read... done
check_allocations... done
journal_replay... done
check_alloc_info... done
check_lrus... done
check_btree_backpointers... done
check_backpointers_to_extents... done
check_extents_to_backpointers... done
check_alloc_to_lru_refs... done
check_snapshot_trees... done
check_snapshots... done
check_subvols... done
delete_dead_snapshots... done
resume_logged_ops... done
check_inodes... done
check_extents... done
check_indirect_extents... done
check_dirents... done
check_xattrs... done
check_root... done
check_directory_structure... done
check_nlinks... done
delete_dead_inodes... done
going read-write
shutting down
flushing journal and stopping allocators, journal seq 1508
flushing journal and stopping allocators complete, journal seq 1508
marking filesystem clean
shutdown complete

After this mount the FS and read the file again, the same errors appear.

koverstreet commented 10 months ago

yes, fsck is only for checking/repairing metadata, you want scrub