search

koverstreet / bcachefs

Other
689 stars 72 forks source link

fsck doesn't do anything after "fix? (y,n,Y,N)", bcachefs won't mount a4643fd #549

Closed trougnouf closed 1 year ago

trougnouf commented 1 year ago

Version

Generic info


$ sudo bcachefs fs usage /dev/nvme2n1
error opening /dev/nvme2n1: not a bcachefs filesystem
$ sudo bcachefs show-super /dev/sda ``` External UUID: 9d883cf4-3574-4d4d-a74f-45457bd07cac Internal UUID: e871d6f5-0d10-4427-9f2b-577f294cfb94 Device index: 3 Label: Version: snapshot_trees Oldest version on disk: fragmentation_lru Created: Tue Mar 7 18:22:51 2023 Sequence number: 140 Superblock size: 6488 Clean: 0 Devices: 5 Sections: members,replicas_v0,disk_groups,clean,journal_seq_blacklist,journal_v2,counters Features: lz4,journal_seq_blacklist_v3,reflink,new_siphash,inline_data,new_extent_overwrite,btree_ptr_v2,extents_above_btree_updates,btree_updates_journalled,reflink_inline_data,new_varint,journal_no_flush,alloc_v2,extents_across_btree_nodes Compat features: alloc_info,alloc_metadata,extents_above_btree_updates_done,bformat_overflow_done Options: block_size: 4.00 KiB btree_node_size: 256 KiB errors: continue [ro] panic metadata_replicas: 3 data_replicas: 2 metadata_replicas_required: 1 data_replicas_required: 1 encoded_extent_max: 64.0 KiB metadata_checksum: none [crc32c] crc64 xxhash data_checksum: none [crc32c] crc64 xxhash compression: none [lz4] gzip zstd background_compression: [none] lz4 gzip zstd str_hash: crc32c crc64 [siphash] metadata_target: none foreground_target: ssd background_target: hdd promote_target: ssd erasure_code: 0 inodes_32bit: 1 shard_inode_numbers: 1 inodes_use_key_cache: 1 gc_reserve_percent: 8 gc_reserve_bytes: 0 B root_reserve_percent: 0 wide_macs: 0 acl: 1 usrquota: 0 grpquota: 0 prjquota: 0 journal_flush_delay: 1000 journal_flush_disabled: 0 journal_reclaim_delay: 100 journal_transaction_names: 1 nocow: 0 members (size 288): Device: 0 UUID: ccd2ec0f-8c27-4a5c-92f3-cd289e1945c9 Size: 954 GiB Bucket size: 512 KiB First bucket: 0 Buckets: 1953524 Last mount: Sat Jun 3 08:23:07 2023 State: rw Label: ssd1 (1) Data allowed: journal,btree,user Has data: journal,btree,user,cached Discard: 0 Freespace initialized: 1 Device: 1 UUID: 9557b304-37f1-4638-a7f4-7841c8e2e490 Size: 14.6 TiB Bucket size: 512 KiB First bucket: 0 Buckets: 30519296 Last mount: Sat Jun 3 08:23:07 2023 State: rw Label: hdd16tb1 (3) Data allowed: journal,btree,user Has data: journal,btree,user,cached Discard: 0 Freespace initialized: 1 Device: 2 UUID: 410ed87f-c09f-4caf-a841-e41bb0576745 Size: 14.6 TiB Bucket size: 512 KiB First bucket: 0 Buckets: 30519296 Last mount: Sat Jun 3 08:23:07 2023 State: rw Label: hdd16tb2 (4) Data allowed: journal,btree,user Has data: journal,btree,user,cached Discard: 0 Freespace initialized: 1 Device: 3 UUID: 6289f799-0c98-4aff-941d-bbd0f5f34b3b Size: 7.28 TiB Bucket size: 512 KiB First bucket: 0 Buckets: 15261770 Last mount: Sat Jun 3 08:23:07 2023 State: rw Label: hdd8tb1 (5) Data allowed: journal,btree,user Has data: journal,btree,user,cached Discard: 0 Freespace initialized: 1 Device: 4 UUID: 2f4111ac-43e6-4e76-8087-9772543c706a Size: 1.82 TiB Bucket size: 1.00 MiB First bucket: 0 Buckets: 1907729 Last mount: Sat Jun 3 08:23:07 2023 State: rw Label: ssd2 (7) Data allowed: journal,btree,user Has data: journal,btree,user,cached Discard: 0 Freespace initialized: 1 ```
$ sudo bcachefs show-super /dev/sdb ``` External UUID: 9d883cf4-3574-4d4d-a74f-45457bd07cac Internal UUID: e871d6f5-0d10-4427-9f2b-577f294cfb94 Device index: 1 Label: Version: snapshot_trees Oldest version on disk: fragmentation_lru Created: Tue Mar 7 18:22:51 2023 Sequence number: 140 Superblock size: 6488 Clean: 0 Devices: 5 Sections: members,replicas_v0,disk_groups,clean,journal_seq_blacklist,journal_v2,counters Features: lz4,journal_seq_blacklist_v3,reflink,new_siphash,inline_data,new_extent_overwrite,btree_ptr_v2,extents_above_btree_updates,btree_updates_journalled,reflink_inline_data,new_varint,journal_no_flush,alloc_v2,extents_across_btree_nodes Compat features: alloc_info,alloc_metadata,extents_above_btree_updates_done,bformat_overflow_done Options: block_size: 4.00 KiB btree_node_size: 256 KiB errors: continue [ro] panic metadata_replicas: 3 data_replicas: 2 metadata_replicas_required: 1 data_replicas_required: 1 encoded_extent_max: 64.0 KiB metadata_checksum: none [crc32c] crc64 xxhash data_checksum: none [crc32c] crc64 xxhash compression: none [lz4] gzip zstd background_compression: [none] lz4 gzip zstd str_hash: crc32c crc64 [siphash] metadata_target: none foreground_target: ssd background_target: hdd promote_target: ssd erasure_code: 0 inodes_32bit: 1 shard_inode_numbers: 1 inodes_use_key_cache: 1 gc_reserve_percent: 8 gc_reserve_bytes: 0 B root_reserve_percent: 0 wide_macs: 0 acl: 1 usrquota: 0 grpquota: 0 prjquota: 0 journal_flush_delay: 1000 journal_flush_disabled: 0 journal_reclaim_delay: 100 journal_transaction_names: 1 nocow: 0 members (size 288): Device: 0 UUID: ccd2ec0f-8c27-4a5c-92f3-cd289e1945c9 Size: 954 GiB Bucket size: 512 KiB First bucket: 0 Buckets: 1953524 Last mount: Sat Jun 3 08:23:07 2023 State: rw Label: ssd1 (1) Data allowed: journal,btree,user Has data: journal,btree,user,cached Discard: 0 Freespace initialized: 1 Device: 1 UUID: 9557b304-37f1-4638-a7f4-7841c8e2e490 Size: 14.6 TiB Bucket size: 512 KiB First bucket: 0 Buckets: 30519296 Last mount: Sat Jun 3 08:23:07 2023 State: rw Label: hdd16tb1 (3) Data allowed: journal,btree,user Has data: journal,btree,user,cached Discard: 0 Freespace initialized: 1 Device: 2 UUID: 410ed87f-c09f-4caf-a841-e41bb0576745 Size: 14.6 TiB Bucket size: 512 KiB First bucket: 0 Buckets: 30519296 Last mount: Sat Jun 3 08:23:07 2023 State: rw Label: hdd16tb2 (4) Data allowed: journal,btree,user Has data: journal,btree,user,cached Discard: 0 Freespace initialized: 1 Device: 3 UUID: 6289f799-0c98-4aff-941d-bbd0f5f34b3b Size: 7.28 TiB Bucket size: 512 KiB First bucket: 0 Buckets: 15261770 Last mount: Sat Jun 3 08:23:07 2023 State: rw Label: hdd8tb1 (5) Data allowed: journal,btree,user Has data: journal,btree,user,cached Discard: 0 Freespace initialized: 1 Device: 4 UUID: 2f4111ac-43e6-4e76-8087-9772543c706a Size: 1.82 TiB Bucket size: 1.00 MiB First bucket: 0 Buckets: 1907729 Last mount: Sat Jun 3 08:23:07 2023 State: rw Label: ssd2 (7) Data allowed: journal,btree,user Has data: journal,btree,user,cached Discard: 0 Freespace initialized: 1 ```
$ sudo bcachefs show-super /dev/sdc ``` External UUID: 9d883cf4-3574-4d4d-a74f-45457bd07cac Internal UUID: e871d6f5-0d10-4427-9f2b-577f294cfb94 Device index: 2 Label: Version: snapshot_trees Oldest version on disk: fragmentation_lru Created: Tue Mar 7 18:22:51 2023 Sequence number: 140 Superblock size: 6488 Clean: 0 Devices: 5 Sections: members,replicas_v0,disk_groups,clean,journal_seq_blacklist,journal_v2,counters Features: lz4,journal_seq_blacklist_v3,reflink,new_siphash,inline_data,new_extent_overwrite,btree_ptr_v2,extents_above_btree_updates,btree_updates_journalled,reflink_inline_data,new_varint,journal_no_flush,alloc_v2,extents_across_btree_nodes Compat features: alloc_info,alloc_metadata,extents_above_btree_updates_done,bformat_overflow_done Options: block_size: 4.00 KiB btree_node_size: 256 KiB errors: continue [ro] panic metadata_replicas: 3 data_replicas: 2 metadata_replicas_required: 1 data_replicas_required: 1 encoded_extent_max: 64.0 KiB metadata_checksum: none [crc32c] crc64 xxhash data_checksum: none [crc32c] crc64 xxhash compression: none [lz4] gzip zstd background_compression: [none] lz4 gzip zstd str_hash: crc32c crc64 [siphash] metadata_target: none foreground_target: ssd background_target: hdd promote_target: ssd erasure_code: 0 inodes_32bit: 1 shard_inode_numbers: 1 inodes_use_key_cache: 1 gc_reserve_percent: 8 gc_reserve_bytes: 0 B root_reserve_percent: 0 wide_macs: 0 acl: 1 usrquota: 0 grpquota: 0 prjquota: 0 journal_flush_delay: 1000 journal_flush_disabled: 0 journal_reclaim_delay: 100 journal_transaction_names: 1 nocow: 0 members (size 288): Device: 0 UUID: ccd2ec0f-8c27-4a5c-92f3-cd289e1945c9 Size: 954 GiB Bucket size: 512 KiB First bucket: 0 Buckets: 1953524 Last mount: Sat Jun 3 08:23:07 2023 State: rw Label: ssd1 (1) Data allowed: journal,btree,user Has data: journal,btree,user,cached Discard: 0 Freespace initialized: 1 Device: 1 UUID: 9557b304-37f1-4638-a7f4-7841c8e2e490 Size: 14.6 TiB Bucket size: 512 KiB First bucket: 0 Buckets: 30519296 Last mount: Sat Jun 3 08:23:07 2023 State: rw Label: hdd16tb1 (3) Data allowed: journal,btree,user Has data: journal,btree,user,cached Discard: 0 Freespace initialized: 1 Device: 2 UUID: 410ed87f-c09f-4caf-a841-e41bb0576745 Size: 14.6 TiB Bucket size: 512 KiB First bucket: 0 Buckets: 30519296 Last mount: Sat Jun 3 08:23:07 2023 State: rw Label: hdd16tb2 (4) Data allowed: journal,btree,user Has data: journal,btree,user,cached Discard: 0 Freespace initialized: 1 Device: 3 UUID: 6289f799-0c98-4aff-941d-bbd0f5f34b3b Size: 7.28 TiB Bucket size: 512 KiB First bucket: 0 Buckets: 15261770 Last mount: Sat Jun 3 08:23:07 2023 State: rw Label: hdd8tb1 (5) Data allowed: journal,btree,user Has data: journal,btree,user,cached Discard: 0 Freespace initialized: 1 Device: 4 UUID: 2f4111ac-43e6-4e76-8087-9772543c706a Size: 1.82 TiB Bucket size: 1.00 MiB First bucket: 0 Buckets: 1907729 Last mount: Sat Jun 3 08:23:07 2023 State: rw Label: ssd2 (7) Data allowed: journal,btree,user Has data: journal,btree,user,cached Discard: 0 Freespace initialized: 1 ```
$ sudo bcachefs show-super /dev/nvme0n1 ``` External UUID: 9d883cf4-3574-4d4d-a74f-45457bd07cac Internal UUID: e871d6f5-0d10-4427-9f2b-577f294cfb94 Device index: 4 Label: Version: snapshot_trees Oldest version on disk: fragmentation_lru Created: Tue Mar 7 18:22:51 2023 Sequence number: 140 Superblock size: 6472 Clean: 0 Devices: 5 Sections: members,replicas_v0,disk_groups,clean,journal_seq_blacklist,journal_v2,counters Features: lz4,journal_seq_blacklist_v3,reflink,new_siphash,inline_data,new_extent_overwrite,btree_ptr_v2,extents_above_btree_updates,btree_updates_journalled,reflink_inline_data,new_varint,journal_no_flush,alloc_v2,extents_across_btree_nodes Compat features: alloc_info,alloc_metadata,extents_above_btree_updates_done,bformat_overflow_done Options: block_size: 4.00 KiB btree_node_size: 256 KiB errors: continue [ro] panic metadata_replicas: 3 data_replicas: 2 metadata_replicas_required: 1 data_replicas_required: 1 encoded_extent_max: 64.0 KiB metadata_checksum: none [crc32c] crc64 xxhash data_checksum: none [crc32c] crc64 xxhash compression: none [lz4] gzip zstd background_compression: [none] lz4 gzip zstd str_hash: crc32c crc64 [siphash] metadata_target: none foreground_target: ssd background_target: hdd promote_target: ssd erasure_code: 0 inodes_32bit: 1 shard_inode_numbers: 1 inodes_use_key_cache: 1 gc_reserve_percent: 8 gc_reserve_bytes: 0 B root_reserve_percent: 0 wide_macs: 0 acl: 1 usrquota: 0 grpquota: 0 prjquota: 0 journal_flush_delay: 1000 journal_flush_disabled: 0 journal_reclaim_delay: 100 journal_transaction_names: 1 nocow: 0 members (size 288): Device: 0 UUID: ccd2ec0f-8c27-4a5c-92f3-cd289e1945c9 Size: 954 GiB Bucket size: 512 KiB First bucket: 0 Buckets: 1953524 Last mount: Sat Jun 3 08:23:07 2023 State: rw Label: ssd1 (1) Data allowed: journal,btree,user Has data: journal,btree,user,cached Discard: 0 Freespace initialized: 1 Device: 1 UUID: 9557b304-37f1-4638-a7f4-7841c8e2e490 Size: 14.6 TiB Bucket size: 512 KiB First bucket: 0 Buckets: 30519296 Last mount: Sat Jun 3 08:23:07 2023 State: rw Label: hdd16tb1 (3) Data allowed: journal,btree,user Has data: journal,btree,user,cached Discard: 0 Freespace initialized: 1 Device: 2 UUID: 410ed87f-c09f-4caf-a841-e41bb0576745 Size: 14.6 TiB Bucket size: 512 KiB First bucket: 0 Buckets: 30519296 Last mount: Sat Jun 3 08:23:07 2023 State: rw Label: hdd16tb2 (4) Data allowed: journal,btree,user Has data: journal,btree,user,cached Discard: 0 Freespace initialized: 1 Device: 3 UUID: 6289f799-0c98-4aff-941d-bbd0f5f34b3b Size: 7.28 TiB Bucket size: 512 KiB First bucket: 0 Buckets: 15261770 Last mount: Sat Jun 3 08:23:07 2023 State: rw Label: hdd8tb1 (5) Data allowed: journal,btree,user Has data: journal,btree,user,cached Discard: 0 Freespace initialized: 1 Device: 4 UUID: 2f4111ac-43e6-4e76-8087-9772543c706a Size: 1.82 TiB Bucket size: 1.00 MiB First bucket: 0 Buckets: 1907729 Last mount: Sat Jun 3 08:23:07 2023 State: rw Label: ssd2 (7) Data allowed: journal,btree,user Has data: journal,btree,user,cached Discard: 0 Freespace initialized: 1 ```
$ sudo bcachefs show-super /dev/nvme2n1 ``` External UUID: 9d883cf4-3574-4d4d-a74f-45457bd07cac Internal UUID: e871d6f5-0d10-4427-9f2b-577f294cfb94 Device index: 0 Label: Version: snapshot_trees Oldest version on disk: fragmentation_lru Created: Tue Mar 7 18:22:51 2023 Sequence number: 140 Superblock size: 6488 Clean: 0 Devices: 5 Sections: members,replicas_v0,disk_groups,clean,journal_seq_blacklist,journal_v2,counters Features: lz4,journal_seq_blacklist_v3,reflink,new_siphash,inline_data,new_extent_overwrite,btree_ptr_v2,extents_above_btree_updates,btree_updates_journalled,reflink_inline_data,new_varint,journal_no_flush,alloc_v2,extents_across_btree_nodes Compat features: alloc_info,alloc_metadata,extents_above_btree_updates_done,bformat_overflow_done Options: block_size: 4.00 KiB btree_node_size: 256 KiB errors: continue [ro] panic metadata_replicas: 3 data_replicas: 2 metadata_replicas_required: 1 data_replicas_required: 1 encoded_extent_max: 64.0 KiB metadata_checksum: none [crc32c] crc64 xxhash data_checksum: none [crc32c] crc64 xxhash compression: none [lz4] gzip zstd background_compression: [none] lz4 gzip zstd str_hash: crc32c crc64 [siphash] metadata_target: none foreground_target: ssd background_target: hdd promote_target: ssd erasure_code: 0 inodes_32bit: 1 shard_inode_numbers: 1 inodes_use_key_cache: 1 gc_reserve_percent: 8 gc_reserve_bytes: 0 B root_reserve_percent: 0 wide_macs: 0 acl: 1 usrquota: 0 grpquota: 0 prjquota: 0 journal_flush_delay: 1000 journal_flush_disabled: 0 journal_reclaim_delay: 100 journal_transaction_names: 1 nocow: 0 members (size 288): Device: 0 UUID: ccd2ec0f-8c27-4a5c-92f3-cd289e1945c9 Size: 954 GiB Bucket size: 512 KiB First bucket: 0 Buckets: 1953524 Last mount: Sat Jun 3 08:23:07 2023 State: rw Label: ssd1 (1) Data allowed: journal,btree,user Has data: journal,btree,user,cached Discard: 0 Freespace initialized: 1 Device: 1 UUID: 9557b304-37f1-4638-a7f4-7841c8e2e490 Size: 14.6 TiB Bucket size: 512 KiB First bucket: 0 Buckets: 30519296 Last mount: Sat Jun 3 08:23:07 2023 State: rw Label: hdd16tb1 (3) Data allowed: journal,btree,user Has data: journal,btree,user,cached Discard: 0 Freespace initialized: 1 Device: 2 UUID: 410ed87f-c09f-4caf-a841-e41bb0576745 Size: 14.6 TiB Bucket size: 512 KiB First bucket: 0 Buckets: 30519296 Last mount: Sat Jun 3 08:23:07 2023 State: rw Label: hdd16tb2 (4) Data allowed: journal,btree,user Has data: journal,btree,user,cached Discard: 0 Freespace initialized: 1 Device: 3 UUID: 6289f799-0c98-4aff-941d-bbd0f5f34b3b Size: 7.28 TiB Bucket size: 512 KiB First bucket: 0 Buckets: 15261770 Last mount: Sat Jun 3 08:23:07 2023 State: rw Label: hdd8tb1 (5) Data allowed: journal,btree,user Has data: journal,btree,user,cached Discard: 0 Freespace initialized: 1 Device: 4 UUID: 2f4111ac-43e6-4e76-8087-9772543c706a Size: 1.82 TiB Bucket size: 1.00 MiB First bucket: 0 Buckets: 1907729 Last mount: Sat Jun 3 08:23:07 2023 State: rw Label: ssd2 (7) Data allowed: journal,btree,user Has data: journal,btree,user,cached Discard: 0 Freespace initialized: 1 ```

Tools bugs

My bcachefs won't mount after an unclean shutdown. fsck appears to ask if I want to have it fixed but the prompt "fix? (y,n,Y,N)" responds to nothing. Below are the relevant logs:

$ sudo mount.bcachefs UUID=9d883cf4-3574-4d4d-a74f-45457bd07cac /orb -o rw,relatime,compression=lz4,metadata_replicas=3,data_replicas=2,foreground_target=ssd,background_target=hdd,promote_target=ssd
ERROR - bcachefs_rust::cmd_mount: Fatal error: No such file or directory
bcachefs (/dev/nvme1n1): error reading default superblock: Not a bcachefs superblockbcachefs (/dev/nvme1n1): error reading superblock: Not a bcachefs superblock layoutbcachefs (/dev/nvme1n1p1): error reading default superblock: Not a bcachefs superblockbcachefs (/dev/nvme1n1p1): error reading superblock: Not a bcachefs superblock layoutbcachefs (/dev/nvme1n1p2): error reading default superblock: Not a bcachefs superblockbcachefs (/dev/nvme1n1p2): error reading superblock: Not a bcachefs superblock layout
dmesg after the above mount command ``` [ 171.900299] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): recovering from unclean shutdown [ 191.785409] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): ja->sectors_free == ca->mi.bucket_size [ 191.785414] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): cur_idx 0/8192 [ 191.785417] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): bucket_seq[8191] = 6845537 [ 191.785419] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): bucket_seq[0] = 6845729 [ 191.785421] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): bucket_seq[1] = 7073927 [ 191.785436] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): journal read done, replaying entries 7073963-7073963 [ 191.785442] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): dropped unflushed entries 7073964-7073964 [ 192.170995] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): going read-write [ 193.181435] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): journal replay done [ 193.181456] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): error -2 checking snapshots [ 193.181504] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): Error in recovery: error in recovery (ENOENT) [ 193.181506] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): error starting filesystem: ENOENT ```
$ sudo bcachefs fsck /dev/sda /dev/sdb /dev/sdc /dev/nvme0n1 /dev/nvme2n1 # this is what I'm reporting here, though any help getting my filesystem running would be very greatly appreciated ``` recovering from unclean shutdown ja->sectors_free == ca->mi.bucket_size cur_idx 0/8192 bucket_seq[8191] = 6845473 bucket_seq[0] = 6845665 bucket_seq[1] = 7073918 ja->sectors_free == ca->mi.bucket_size cur_idx 0/8192 bucket_seq[8191] = 6845537 bucket_seq[0] = 6845729 bucket_seq[1] = 7073927 journal read done, replaying entries 7073994-7073994 checking allocations starting journal replay, 0 keys journal replay done checking need_discard and freespace btrees checking lrus incorrect lru entry: lru read time 88971037880 u64s 5 type set 1125988877880504:1125899908132210:0 len 0 ver 0 for u64s 12 type alloc_v4 4:1289586:0 len 0 ver 0: gen 2 oldest_gen 0 data_type cached journal_seq 5745080 need_discard 1 need_inc_gen 1 dirty_sectors 0 cached_sectors 2048 stripe 0 stripe_redundancy 0 io_time[READ] 96298116000 io_time[WRITE] 112457381816 fragmentation 0 bp_start 7 : fix? (y,n,Y,N) Y (y,n,Y,N) y (y,n,Y,N) yes (y,n,Y,N) Yes (y,n,Y,N) (y,n,Y,N) n (y,n,Y,N) N (y,n,Y,N) fix (y,n,Y,N) fix? (y,n,Y,N) ^C ```
dmesg after the above fsck: ``` [ 1884.589261] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): recovering from unclean shutdown [ 1900.793094] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): ja->sectors_free == ca->mi.bucket_size [ 1900.793097] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): cur_idx 0/8192 [ 1900.793099] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): bucket_seq[8191] = 6845473 [ 1900.793100] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): bucket_seq[0] = 6845665 [ 1900.793101] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): bucket_seq[1] = 7073918 [ 1904.216762] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): ja->sectors_free == ca->mi.bucket_size [ 1904.216764] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): cur_idx 0/8192 [ 1904.216766] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): bucket_seq[8191] = 6845537 [ 1904.216767] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): bucket_seq[0] = 6845729 [ 1904.216768] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): bucket_seq[1] = 7073927 [ 1904.216781] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): journal read done, replaying entries 7073994-7073994 [ 1904.677054] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): journal replay done [ 1904.677066] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): error -2 checking snapshots [ 1904.677093] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): Error in recovery: error in recovery (ENOENT) [ 1904.677094] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): error starting filesystem: ENOENT ```
# gdb -ex run --args bcachefs fsck /dev/sda /dev/sdb /dev/sdc /dev/nvme0n1 /dev/nvme2n1 ``` GNU gdb (GDB) 13.1 Copyright (C) 2023 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-pc-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: . Find the GDB manual and other documentation resources online at: . For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from bcachefs... Starting program: /usr/bin/bcachefs fsck /dev/sda /dev/sdb /dev/sdc /dev/nvme0n1 /dev/nvme2n1 This GDB supports auto-downloading debuginfo from the following URLs: Enable debuginfod for this session? (y or [n]) y Debuginfod has been enabled. To make this setting permanent, add 'set debuginfod enabled on' to .gdbinit. Downloading separate debug info for /lib64/ld-linux-x86-64.so.2 Downloading separate debug info for system-supplied DSO at 0x7ffff7fc8000 Downloading separate debug info for /usr/lib/libblkid.so.1 Downloading separate debug info for /usr/lib/libuuid.so.1 Downloading separate debug info for /usr/lib/liburcu-common.so.8 Downloading separate debug info for /usr/lib/liburcu.so.8 Downloading separate debug info for /usr/lib/libsodium.so.23 Downloading separate debug info for /usr/lib/libz.so.1 Downloading separate debug info for /usr/lib/liblz4.so.1 Downloading separate debug info for /usr/lib/libzstd.so.1 Downloading separate debug info for /usr/lib/libudev.so.1 Downloading separate debug info for /usr/lib/libkeyutils.so.1 Downloading separate debug info for /usr/lib/libm.so.6 Downloading separate debug info for /usr/lib/libaio.so.1 Downloading separate debug info for /usr/lib/libc.so.6 [Thread debugging using libthread_db enabled] Using host libthread_db library "/usr/lib/libthread_db.so.1". Downloading separate debug info for /usr/lib/libpthread.so.0 Downloading separate debug info for /usr/lib/libcap.so.2 [New Thread 0x7ffff7fbe680 (LWP 64605)] [New Thread 0x7ffff7fb5680 (LWP 64606)] [New Thread 0x7ffff7fac680 (LWP 64607)] [New Thread 0x7ffff7fa3680 (LWP 64608)] [New Thread 0x7ffff7f9a680 (LWP 64609)] [New Thread 0x7ffff7a79680 (LWP 64610)] [New Thread 0x7ffff7a70680 (LWP 64611)] [New Thread 0x7ffff7a67680 (LWP 64612)] [New Thread 0x7ffff79dc680 (LWP 64613)] [New Thread 0x7ffff79d3680 (LWP 64614)] [New Thread 0x7ffff79ca680 (LWP 64615)] [New Thread 0x7ffff79c1680 (LWP 64616)] [New Thread 0x7ffff79b8680 (LWP 64617)] [New Thread 0x7ffff6f0d680 (LWP 64618)] recovering from unclean shutdown ja->sectors_free == ca->mi.bucket_size cur_idx 0/8192 bucket_seq[8191] = 6845473 bucket_seq[0] = 6845665 bucket_seq[1] = 7073918 ja->sectors_free == ca->mi.bucket_size cur_idx 0/8192 bucket_seq[8191] = 6845537 bucket_seq[0] = 6845729 bucket_seq[1] = 7073927 journal read done, replaying entries 7073994-7073994 [New Thread 0x7ffff1974680 (LWP 64782)] [New Thread 0x7ffff196b680 (LWP 64783)] checking allocations starting journal replay, 0 keys journal replay done checking need_discard and freespace btrees checking lrus incorrect lru entry: lru read time 88971037880 u64s 5 type set 1125988877880504:1125899908132210:0 len 0 ver 0 for u64s 12 type alloc_v4 4:1289586:0 len 0 ver 0: gen 2 oldest_gen 0 data_type cached journal_seq 5745080 need_discard 1 need_inc_gen 1 dirty_sectors 0 cached_sectors 2048 stripe 0 stripe_redundancy 0 io_time[READ] 96298116000 io_time[WRITE] 112457381816 fragmentation 0 bp_start 7 : fix? (y,n,Y,N) Y (y,n,Y,N) y (y,n,Y,N) yes (y,n,Y,N) ? (y,n,Y,N) f (y,n,Y,N) 1 (y,n,Y,N) 0 (y,n,Y,N) T (y,n,Y,N) ^C Thread 1 "bcachefs" received signal SIGINT, Interrupt. 0x00007ffff7b89b5c in __GI___libc_read (nbytes=1024, buf=0x555555c23400, fd=0) at ../sysdeps/unix/sysv/linux/read.c:26 Downloading source file /usr/src/debug/glibc/glibc/io/../sysdeps/unix/sysv/linux/read.c [### [ ### 26 return SYSCALL_CANCEL (read, fd, buf, nbytes); (gdb) bt #0 0x00007ffff7b89b5c in __GI___libc_read (nbytes=1024, buf=0x555555c23400, fd=0) at ../sysdeps/unix/sysv/linux/read.c:26 #1 __GI___libc_read (fd=0, buf=0x555555c23400, nbytes=1024) at ../sysdeps/unix/sysv/linux/read.c:24 #2 0x00007ffff7b0fa04 in _IO_new_file_underflow (fp=0x7ffff7c6b9c0 <_IO_2_1_stdin_>) at /usr/src/debug/glibc/glibc/libio/libioP.h:946 #3 0x00007ffff7b03670 in __GI___getdelim (lineptr=lineptr@entry=0x7fffffffd478, n=n@entry=0x7fffffffd480, delimiter=delimiter@entry=10, fp=0x7ffff7c6b9c0 <_IO_2_1_stdin_>) at iogetdelim.c:77 #4 0x0000555555617c58 in getline (__stream=, __n=0x7fffffffd480, __lineptr=0x7fffffffd478) at /usr/include/bits/stdio.h:120 #5 bch2_fsck_ask_yn () at libbcachefs/error.c:91 #6 0x00005555556185a3 in bch2_fsck_err (c=c@entry=0x7ffff79de000, flags=flags@entry=3, fmt=fmt@entry=0x55555584c4d0 "incorrect lru entry: lru %s time %llu\n %s\n for %s") at libbcachefs/error.c:221 #7 0x000055555564be1f in bch2_check_lru_key (last_flushed_pos=, lru_k=..., lru_iter=0x7fffffffd760, trans=0x7fffffffd880) at libbcachefs/lru.c:143 #8 bch2_check_lrus (c=c@entry=0x7ffff79de000) at libbcachefs/lru.c:171 #9 0x000055555565a539 in bch2_fs_recovery (c=) at libbcachefs/recovery.c:1355 #10 0x000055555566f45e in bch2_fs_start (c=c@entry=0x7ffff79de000) at libbcachefs/super.c:934 #11 0x00005555556724ba in bch2_fs_open (devices=devices@entry=0x7fffffffe3c0, nr_devices=, nr_devices@entry=5, opts=...) at libbcachefs/super.c:1924 #12 0x00005555555a3ee8 in cmd_fsck (argc=5, argv=0x7fffffffe3c0, argv@entry=0x7fffffffe3b8) at cmd_fsck.c:96 #13 0x000055555559c3f5 in main (argc=, argv=0x7fffffffe3b8) at bcachefs.c:205 ```
trougnouf commented 1 year ago

The issue, besides the FS corruption and inability to mount, seems to be the input handling as described in the title. Invoking "bcachefs fsck -p " does fix

a whole lot of these "incorrect lru entry: lru read time" errors ``` incorrect lru entry: lru read time 88997581456 u64s 5 type set 1125988904424080:1125899908042185:0 len 0 ver 0 for u64s 12 type alloc_v4 4:1199561:0 len 0 ver 0: gen 2 oldest_gen 0 data_type cached journal_seq 5745731 need_discard 1 need_inc_gen 1 dirty_sectors 0 cached_sectors 2048 stripe 0 stripe_redundancy 0 io_time[READ] 96279964832 io_time[WRITE] 112479880184 fragmentation 0 bp_start 7 , fixing incorrect lru entry: lru read time 88997585432 u64s 5 type set 1125988904428056:1125899908042191:0 len 0 ver 0 for u64s 12 type alloc_v4 4:1199567:0 len 0 ver 0: gen 2 oldest_gen 0 data_type cached journal_seq 5745731 need_discard 1 need_inc_gen 1 dirty_sectors 0 cached_sectors 2048 stripe 0 stripe_redundancy 0 io_time[READ] 96279912240 io_time[WRITE] 112479883584 fragmentation 0 bp_start 7 , fixing incorrect lru entry: lru read time 88997587888 u64s 5 type set 1125988904430512:1125899908042192:0 len 0 ver 0 for u64s 12 type alloc_v4 4:1199568:0 len 0 ver 0: gen 2 oldest_gen 0 data_type cached journal_seq 5745731 need_discard 1 need_inc_gen 1 dirty_sectors 0 cached_sectors 2048 stripe 0 stripe_redundancy 0 io_time[READ] 96279919304 io_time[WRITE] 112479885480 fragmentation 0 bp_start 7 , fixing ```

I ended up giving up on bcachefs fsck because it's too slow and it seems that mount.bcachefs -o ...,fsck,fix_errors works

with the following fixed errors in dmesg

[Jun 4 23:19] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): recovering from unclean shutdown
[ +17.612837] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): ja->sectors_free == ca->mi.bucket_size
[  +0.000004] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): cur_idx 0/8192
[  +0.000003] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): bucket_seq[8191] = 6845537
[  +0.000003] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): bucket_seq[0] = 6845729
[  +0.000001] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): bucket_seq[1] = 7073927
[  +0.000031] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): journal read done, replaying entries 7074003-7074078
[  +0.398225] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): checking allocations
[Jun 4 23:34] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): starting journal replay, 4905 keys
[  +0.000540] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): going read-write
[  +3.759552] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): journal replay done
[  +0.000005] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): checking need_discard and freespace btrees
[ +21.379713] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): checking lrus
[  +6.358593] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): checking backpointers to alloc keys
[Jun 4 23:46] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): checking backpointers to extents
[Jun 4 23:57] backpointer for missing extent
[  +0.000002]   bucket=1:16186272:0 btree=extents l=0 offset=128:0 len=128 pos=604149741:15672:U32_MAX, fixing
[  +0.000016] backpointer for missing extent
[  +0.000001]   bucket=1:16186272:0 btree=extents l=0 offset=256:0 len=128 pos=604149741:15800:U32_MAX, fixing
[  +0.000005] backpointer for missing extent
[  +0.000000]   bucket=1:16186272:0 btree=extents l=0 offset=384:0 len=80 pos=604149741:15880:U32_MAX, fixing
[  +0.000004] backpointer for missing extent
[  +0.000000]   bucket=1:16186272:0 btree=extents l=0 offset=464:0 len=128 pos=604149741:16008:U32_MAX, fixing
[  +0.000004] backpointer for missing extent
[  +0.000001]   bucket=1:16186272:0 btree=extents l=0 offset=592:0 len=96 pos=604149741:16104:U32_MAX, fixing
[  +0.000004] backpointer for missing extent
[  +0.000000]   bucket=1:16186272:0 btree=extents l=0 offset=688:0 len=32 pos=604149741:16136:U32_MAX, fixing
[  +0.000004] backpointer for missing extent
[  +0.000000]   bucket=1:16186272:0 btree=extents l=0 offset=720:0 len=128 pos=604149741:16264:U32_MAX, fixing
[  +0.000004] backpointer for missing extent
[  +0.000000]   bucket=1:16186272:0 btree=extents l=0 offset=848:0 len=56 pos=604149741:16320:U32_MAX, fixing
[  +0.000004] backpointer for missing extent
[  +0.000000]   bucket=1:16186272:0 btree=extents l=0 offset=904:0 len=120 pos=604149741:16440:U32_MAX, fixing
[Jun 5 00:08] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): checking extents to backpointers
[Jun 5 00:30] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): checking alloc to lru refs
[ +30.554001] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): starting fsck
[  +0.000014] snapshot points to missing/incorrect tree:
[  +0.000001]   u64s 8 type snapshot 0:4294967295:0 len 0 ver 0: is_subvol 1 deleted 0 parent          0 children          0          0 subvol 1, fixing
[Jun 5 00:31] extent in missing inode:
[  +0.000002]   u64s 9 type extent 268715392:103494000:U32_MAX len 8 ver 0: durability: 2 crc: c_size 8 size 8 offset 0 nonce 0 csum crc32c compress incompressible ptr: 2:18030724:784 gen 2 ptr: 1:20366460:992 gen 0 ptr: 3:5840930:1016 gen 5 cached, fixing
[  +0.003539] extent in missing inode:
[  +0.000001]   u64s 9 type extent 268715392:105981824:U32_MAX len 88 ver 0: durability: 2 crc: c_size 88 size 88 offset 0 nonce 0 csum crc32c compress incompressible ptr: 3:6249214:680 gen 0 ptr: 1:20385213:472 gen 0 ptr: 2:877322:936 gen 5 cached, fixing
[  +0.003651] extent in missing inode:
[  +0.000001]   u64s 8 type extent 268715392:113852440:U32_MAX len 56 ver 0: durability: 2 crc: c_size 56 size 56 offset 0 nonce 0 csum crc32c compress incompressible ptr: 2:12647064:0 gen 0 ptr: 1:20377573:224 gen 0, fixing
[  +0.002647] extent in missing inode:
[  +0.000000]   u64s 8 type extent 268715392:115038832:U32_MAX len 24 ver 0: durability: 2 crc: c_size 24 size 24 offset 0 nonce 0 csum crc32c compress incompressible ptr: 2:18362906:520 gen 1 ptr: 1:20371069:744 gen 0, fixing
[  +0.001824] extent in missing inode:
[  +0.000001]   u64s 9 type extent 268715392:116576288:U32_MAX len 88 ver 0: durability: 2 crc: c_size 88 size 88 offset 0 nonce 0 csum crc32c compress incompressible ptr: 3:6249214:768 gen 0 ptr: 1:20385213:560 gen 0 ptr: 2:913362:936 gen 8 cached, fixing
[  +0.002288] extent in missing inode:
[  +0.000001]   u64s 9 type extent 268715392:117130952:U32_MAX len 16 ver 0: durability: 2 crc: c_size 16 size 16 offset 0 nonce 0 csum crc32c compress incompressible ptr: 3:10305754:224 gen 1 ptr: 2:2006431:256 gen 1 cached ptr: 1:20369192:728 gen 0, fixing
[Jun 5 00:33] bcachefs (9d883cf4-3574-4d4d-a74f-45457bd07cac): mounted version=snapshot_trees opts=metadata_replicas=3,data_replicas=2,compression=lz4,foreground_target=/dev/sdb,background_target=/dev/sdb,promote_target=/dev/sdb,fsck,fix_errors

I still get this error from the mount command (but /dev/nvme1* is not part of the bcachefs so it's probably trying every drive / normal) ``` [trougnouf@sd ~]$ sudo mount.bcachefs UUID=9d883cf4-3574-4d4d-a74f-45457bd07cac /orb -o rw,relatime,compression=lz4,metadata_replicas=3,data_replicas=2,foreground_target=ssd,background_target=hdd,promote_target=ssd,fsck,fix_errors [sudo] password for trougnouf: bcachefs (/dev/nvme1n1): error reading default superblock: Not a bcachefs superblockbcachefs (/dev/nvme1n1): error reading superblock: Not a bcachefs superblock layoutbcachefs (/dev/nvme1n1p1): error reading default superblock: Not a bcachefs superblockbcachefs (/dev/nvme1n1p1): error reading superblock: Not a bcachefs superblock layoutbcachefs (/dev/nvme1n1p2): error reading default superblock: Not a bcachefs superblockbcachefs (/dev/nvme1n1p2): error reading superblock: Not a bcachefs superblock layout ```

but I have a working filesystem again! :)

koverstreet commented 1 year ago

I just pushed a fix for the prompt bug - we weren't trimming the newline before checking the string length.

Have not dug into the rest of these yet.

trougnouf commented 1 year ago

Thank you! :) As far as I'm concerned the bug is fixed since I have a working filesystem, I will let you close it in case there's more that caused the errors in the first place. I don't know if I missed anything on the IRC, I was using the web interface and it doesn't show me messages once it disconnects, sorry if that's the case.