Closed Mjasnik closed 10 months ago
Version The last commit for kernel 6.5, I can not find the commit anymore, but this was happening before too.
Generic info
bcachefs fs usage Filesystem: bc9b7455-7f74-47b1-8e78-65a58c3d07db Size: 39513699328 Used: 32986152960 Online reserved: 0 Data type Required/total Devices btree: 1/2 [sda5 sdb5] 179306496 user: 1/2 [sda5 sdb5] 32464486400 sea1.bch (device 0): sda5 rw data buckets fragmented free: 0 19003 sb: 3149824 13 258048 journal: 167772160 640 btree: 89653248 342 user: 16232243200 61922 237568 cached: 0 0 parity: 0 0 stripe: 0 0 need_gc_gens: 0 0 need_discard: 0 0 erasure coded: 0 0 capacity: 21474836480 81920 sea2.bch (device 1): sdb5 rw data buckets fragmented free: 0 19003 sb: 3149824 13 258048 journal: 167772160 640 btree: 89653248 342 user: 16232243200 61922 237568 cached: 0 0 parity: 0 0 stripe: 0 0 need_gc_gens: 0 0 need_discard: 0 0 erasure coded: 0 0 capacity: 21474836480 81920
bcachefs show-super External UUID: bc9b7455-7f74-47b1-8e78-65a58c3d07db Internal UUID: 5ce964f8-d550-434f-8441-5543b29b3fd1 Device index: 0 Label: BIGDATA2.BCH Version: 1.3: rebalance_work Version upgrade complete: 1.3: rebalance_work Oldest version on disk: 1.3: rebalance_work Created: Fri Nov 3 22:52:16 2023 Sequence number: 41 Superblock size: 5088 Clean: 0 Devices: 2 Sections: members_v1,replicas_v0,disk_groups,clean,journal_v2,counters,members_v2,errors Features: new_siphash,inline_data,new_extent_overwrite,btree_ptr_v2,extents_above_btree_updates,btree_updates_journalled,new_varint,journal_no_flush,alloc_v2,extents_across_btree_nodes Compat features: alloc_info,alloc_metadata,extents_above_btree_updates_done,bformat_overflow_done Options: block_size: 4.00 KiB btree_node_size: 256 KiB errors: continue [ro] panic metadata_replicas: 2 data_replicas: 2 metadata_replicas_required: 1 data_replicas_required: 1 encoded_extent_max: 64.0 KiB metadata_checksum: none [crc32c] crc64 xxhash data_checksum: none [crc32c] crc64 xxhash compression: none background_compression: none str_hash: crc32c crc64 [siphash] metadata_target: none foreground_target: none background_target: none promote_target: none erasure_code: 0 inodes_32bit: 1 shard_inode_numbers: 1 inodes_use_key_cache: 1 gc_reserve_percent: 8 gc_reserve_bytes: 0 B root_reserve_percent: 0 wide_macs: 0 acl: 1 usrquota: 0 grpquota: 0 prjquota: 0 journal_flush_delay: 1000 journal_flush_disabled: 0 journal_reclaim_delay: 100 journal_transaction_names: 1 version_upgrade: [compatible] incompatible none nocow: 0 members_v2 (size 256): Device: 0 Label: bch (1) UUID: 5cfeb555-a2dc-4ad1-ac1a-018fdc6340aa Size: 20.0 GiB read errors: 0 write errors: 0 checksum errors: 0 seqread iops: 0 seqwrite iops: 0 randread iops: 0 randwrite iops: 0 Bucket size: 256 KiB First bucket: 0 Buckets: 81920 Last mount: Sat Nov 4 00:35:22 2023 State: rw Data allowed: journal,btree,user Has data: journal,btree,user Discard: 0 Freespace initialized: 1 Device: 1 Label: bch (3) UUID: a5bbcd81-035a-40e7-ad38-47ed168e20b7 Size: 20.0 GiB read errors: 0 write errors: 0 checksum errors: 653 seqread iops: 0 seqwrite iops: 0 randread iops: 0 randwrite iops: 0 Bucket size: 256 KiB First bucket: 0 Buckets: 81920 Last mount: Sat Nov 4 00:35:22 2023 State: rw Data allowed: journal,btree,user Has data: journal,btree,user Discard: 0 Freespace initialized: 1 replicas_v0 (size 24): btree: 2 [0 1] journal: 2 [0 1] user: 2 [0 1]
The issue The raid1 errors do not get detected by fsck, fsck does not find and fix them, there is no scrub or rebalance, how to fix them? The situation is created artificially, sort of simulating huge bitrot.
sudo bcachefs format --replicas=2 -L BIGDATA2.BCH --label=sea1.bch /dev/disk/by-partlabel/sea1.bch1 --label=sea2.bch /dev/disk/by-partlabel/sea2.bch1 sudo mkdir /mnt/BIGDATA2.BCH sudo mount -t bcachefs /dev/disk/by-partlabel/sea1.bch1:/dev/disk/by-partlabel/sea2.bch1 /mnt/BIGDATA2.BCH sudo chown user:user /mnt/BIGDATA2.BCH time rsync --info=progress2 --fsync -a "/mnt/DATA/WinePrefixes/GOG.Galaxy/drive_c/Program Files (x86)/GOG Galaxy/Games/Cyberpunk 2077/archive/pc/content/basegame_4_appearance.archive" /mnt/BIGDATA2.BCH/ sudo umount /mnt/BIGDATA.BCH sudo dd if=/dev/urandom of=/dev/disk/by-partlabel/sea2.bch1 bs=1024000 count=10 oflag=direct seek=10240 sudo mount -t bcachefs /dev/disk/by-partlabel/sea1.bch1:/dev/disk/by-partlabel/sea2.bch1 /mnt/BIGDATA2.BCH
Doing this
time md5sum /mnt/BIGDATA2.BCH/basegame_4_appearance.archive
results in errors in kernel log, md5sum is correct, i.e. it reads the file correctly
nov 04 00:36:47 crazy kernel: bcachefs (sdb5 inum 67108864 offset 10256252928): data checksum error: expected 0:6d7053e4 got 0:4899c936 (type crc32c) ...
Fsck does not help
sudo bcachefs fsck -pvy /dev/disk/by-partlabel/sea2.bch1 /dev/disk/by-partlabel/sea1.bch1mounting version 1.3: rebalance_work opts=metadata_replicas=2,data_replicas=2,degraded,verbose,fsck,fix_errors=yes recovering from clean shutdown, journal seq 1508 starting journal read journal read done on device 0x55f00d8f5750g, ret 0 journal read done on device 0x55f00d8f5380g, ret 0 journal read done, replaying entries 1508-1508 alloc_read... done stripes_read... done snapshots_read... done check_allocations... done journal_replay... done check_alloc_info... done check_lrus... done check_btree_backpointers... done check_backpointers_to_extents... done check_extents_to_backpointers... done check_alloc_to_lru_refs... done check_snapshot_trees... done check_snapshots... done check_subvols... done delete_dead_snapshots... done resume_logged_ops... done check_inodes... done check_extents... done check_indirect_extents... done check_dirents... done check_xattrs... done check_root... done check_directory_structure... done check_nlinks... done delete_dead_inodes... done going read-write shutting down flushing journal and stopping allocators, journal seq 1508 flushing journal and stopping allocators complete, journal seq 1508 marking filesystem clean shutdown complete
After this mount the FS and read the file again, the same errors appear.
yes, fsck is only for checking/repairing metadata, you want scrub
Version The last commit for kernel 6.5, I can not find the commit anymore, but this was happening before too.
Generic info
The issue The raid1 errors do not get detected by fsck, fsck does not find and fix them, there is no scrub or rebalance, how to fix them? The situation is created artificially, sort of simulating huge bitrot.
Doing this
results in errors in kernel log, md5sum is correct, i.e. it reads the file correctly
Fsck does not help
After this mount the FS and read the file again, the same errors appear.