search

koverstreet / bcachefs

Other
662 stars 70 forks source link

No data after unclean reboot - fsck errors #580

Closed Doc-Pixel closed 1 year ago

Doc-Pixel commented 1 year ago

Similar issues are #251 #265 #344

I'm running bcacheFS with two NVMEs and one HDD. After a reboot my bcachefs volume is empty when mounted, bcachefs fs usage shows there is still terabytes of data on the volume. fsck throws invalid bkey format: field too large

My volume is set up as below

bcachefs format --label=ssd.ssd1 /dev/nvme0n1 --label=ssd.ssd2 /dev/nvme1n1 --label=hdd.hdd1 /dev/sdb --foreground_target=ssd --promote_target=ssd --background_target=hdd

Context: I'm running several blockchains on the system (i9, 128GB, 2x 1TB nvme, 16TB HDD). One of the chains (erigon) uses MDBX as a database, which is a COW type of database that uses one file to store the database. My database was about 6.3TB and actively written to. Performance was really slow, erigon team advised not to use MDBX on top of bcacheFS. I deleted the erigon data after. My system became slow and unresponsive, I decided to reset.

After a hard reset I mounted my bcachefs volume and it was empty. mount -t bcachefs /dev/nvme0n1:/dev/nvme1n1:/dev/sdb /data I unmounted and performed fsck

It threw error validating btree node on 0x5639cfac65e0g at btree extents level 0/2 u64s 11 type btree_ptr_v2 1744968794:174456:U32_MAX len 0 ver 0: seq 7b59fb704c6fa19a written 328 min_key 1744968793:2975960945:0 durability: 1 ptr: 0:1266968:512 gen 14 node offset 0 bset u64s 8771: invalid bkey format: field too large

bcachefs fsck /dev/nvme0n1 /dev/nvme1n1 /dev/sdb 
recovering from clean shutdown, journal seq 11445813
journal read done, replaying entries 11445813-11445813
checking allocations
error validating btree node on 0x5639cfac65e0g at btree extents level 0/2
  u64s 11 type btree_ptr_v2 1744968794:174456:U32_MAX len 0 ver 0: seq 7b59fb704c6fa19a written 328 min_key 1744968793:2975960945:0 durability: 1 ptr: 0:1266968:512 gen 14
  node offset 0 bset u64s 8771: invalid bkey format: field too large
starting journal replay, 0 keys
checking need_discard and freespace btrees
checking lrus
checking backpointers to alloc keys
checking backpointers to extents
checking extents to backpointers
checking alloc to lru refs
going read-write
mounted version=snapshot_trees opts=foreground_target=ssd,background_target=hdd,promote_target=ssd,degraded,fsck,fix_errors

Version

bcachefs version
bcachefs tool version v0.1-692-gcfa816b

Make sure you're using a reasonably new version.

Generic info Provide the output of:

bcachefs fs usage

Filesystem: 5c3e0b86-40c3-4bff-8d55-a7de43b9399b
Size:                       15.1 TiB
Used:                       6.29 TiB
Online reserved:                 0 B

Data type       Required/total  Devices
reserved:       1/0                    [] 1.74 TiB
btree:          1/1             [nvme1n1]                   14.3 GiB
btree:          1/1             [nvme0n1]                   14.9 GiB
user:           1/1             [nvme1n1]                    111 GiB
user:           1/1             [nvme0n1]                    111 GiB
user:           1/1             [sdb]                       4.27 TiB
cached:         1/1             [nvme0n1]                    256 GiB
cached:         1/1             [nvme1n1]                    449 GiB

hdd.hdd1 (device 2):             sdb              rw
                                data         buckets    fragmented
  free:                          0 B        21532879
  sb:                       3.00 MiB               7       508 KiB
  journal:                  4.00 GiB            8192
  btree:                         0 B               0
  user:                     4.27 TiB         8978218      9.71 GiB
  cached:                        0 B               0
  parity:                        0 B               0
  stripe:                        0 B               0
  need_gc_gens:                  0 B               0
  need_discard:                  0 B               0
  erasure coded:                 0 B               0
  capacity:                 14.6 TiB        30519296

ssd.ssd1 (device 0):         nvme0n1              rw
                                data         buckets    fragmented
  free:                          0 B         1102535
  sb:                       3.00 MiB               7       508 KiB
  journal:                  4.00 GiB            8192
  btree:                    14.9 GiB           47439      8.23 GiB
  user:                      111 GiB          235158      3.62 GiB
  cached:                    256 GiB          560140
  parity:                        0 B               0
  stripe:                        0 B               0
  need_gc_gens:                  0 B               0
  need_discard:                  0 B              53
  erasure coded:                 0 B               0
  capacity:                  954 GiB         1953524

ssd.ssd2 (device 1):         nvme1n1              rw
                                data         buckets    fragmented
  free:                          0 B          671580
  sb:                       3.00 MiB               7       508 KiB
  journal:                  4.00 GiB            8192
  btree:                    14.3 GiB           45731      8.00 GiB
  user:                      111 GiB          232151      2.77 GiB
  cached:                    449 GiB          995812
  parity:                        0 B               0
  stripe:                        0 B               0
  need_gc_gens:                  0 B               0
  need_discard:                  0 B              51
  erasure coded:                 0 B               0
  capacity:                  954 GiB         1953524

bcachefs show-super

bcachefs show-super /dev/nvme0n1 
External UUID:                              5c3e0b86-40c3-4bff-8d55-a7de43b9399b
Internal UUID:                              200f8f2d-9ca5-432f-921f-991e42aa19e1
Device index:                               0
Label:                                      
Version:                                    snapshot_trees
Oldest version on disk:                     snapshot_trees
Created:                                    Thu Jul 13 21:41:57 2023
Sequence number:                            40
Superblock size:                            5040
Clean:                                      0
Devices:                                    3
Sections:                                   members,replicas_v0,disk_groups,clean,journal_seq_blacklist,journal_v2,counters
Features:                                   journal_seq_blacklist_v3,new_siphash,inline_data,new_extent_overwrite,btree_ptr_v2,extents_above_btree_updates,btree_updates_journalled,new_varint,journal_no_flush,alloc_v2,extents_across_btree_nodes
Compat features:                            alloc_info,alloc_metadata,extents_above_btree_updates_done,bformat_overflow_done

Options:
  block_size:                               4.00 KiB
  btree_node_size:                          256 KiB
  errors:                                   continue [ro] panic 
  metadata_replicas:                        1
  data_replicas:                            1
  metadata_replicas_required:               1
  data_replicas_required:                   1
  encoded_extent_max:                       64.0 KiB
  metadata_checksum:                        none [crc32c] crc64 xxhash 
  data_checksum:                            none [crc32c] crc64 xxhash 
  compression:                              [none] lz4 gzip zstd 
  background_compression:                   [none] lz4 gzip zstd 
  str_hash:                                 crc32c crc64 [siphash] 
  metadata_target:                          none
  foreground_target:                        ssd
  background_target:                        hdd
  promote_target:                           ssd
  erasure_code:                             0
  inodes_32bit:                             1
  shard_inode_numbers:                      1
  inodes_use_key_cache:                     1
  gc_reserve_percent:                       8
  gc_reserve_bytes:                         0 B
  root_reserve_percent:                     0
  wide_macs:                                0
  acl:                                      1
  usrquota:                                 0
  grpquota:                                 0
  prjquota:                                 0
  journal_flush_delay:                      1000
  journal_flush_disabled:                   0
  journal_reclaim_delay:                    100
  journal_transaction_names:                1
  nocow:                                    0

members (size 176):
  Device:                                   0
    UUID:                                   8de13c84-c503-44bc-9ac8-c2732d442708
    Size:                                   954 GiB
    Bucket size:                            512 KiB
    First bucket:                           0
    Buckets:                                1953524
    Last mount:                             Thu Aug  3 19:23:30 2023
    State:                                  rw
    Label:                                  ssd1 (1)
    Data allowed:                           journal,btree,user
    Has data:                               journal,btree,user,cached
    Discard:                                0
    Freespace initialized:                  1
  Device:                                   1
    UUID:                                   2cb0ca1c-d805-4c7c-bbb0-e356bc61e526
    Size:                                   954 GiB
    Bucket size:                            512 KiB
    First bucket:                           0
    Buckets:                                1953524
    Last mount:                             Thu Aug  3 19:23:30 2023
    State:                                  rw
    Label:                                  ssd2 (2)
    Data allowed:                           journal,btree,user
    Has data:                               journal,btree,user,cached
    Discard:                                0
    Freespace initialized:                  1
  Device:                                   2
    UUID:                                   90105af4-ef37-41b0-a77a-01e4a4f7da3e
    Size:                                   14.6 TiB
    Bucket size:                            512 KiB
    First bucket:                           0
    Buckets:                                30519296
    Last mount:                             Thu Aug  3 19:23:30 2023
    State:                                  rw
    Label:                                  hdd1 (4)
    Data allowed:                           journal,btree,user
    Has data:                               user
    Discard:                                0
    Freespace initialized:                  1
bcachefs show-super /dev/nvme1n1 
External UUID:                              5c3e0b86-40c3-4bff-8d55-a7de43b9399b
Internal UUID:                              200f8f2d-9ca5-432f-921f-991e42aa19e1
Device index:                               1
Label:                                      
Version:                                    snapshot_trees
Oldest version on disk:                     snapshot_trees
Created:                                    Thu Jul 13 21:41:57 2023
Sequence number:                            40
Superblock size:                            5040
Clean:                                      0
Devices:                                    3
Sections:                                   members,replicas_v0,disk_groups,clean,journal_seq_blacklist,journal_v2,counters
Features:                                   journal_seq_blacklist_v3,new_siphash,inline_data,new_extent_overwrite,btree_ptr_v2,extents_above_btree_updates,btree_updates_journalled,new_varint,journal_no_flush,alloc_v2,extents_across_btree_nodes
Compat features:                            alloc_info,alloc_metadata,extents_above_btree_updates_done,bformat_overflow_done

Options:
  block_size:                               4.00 KiB
  btree_node_size:                          256 KiB
  errors:                                   continue [ro] panic 
  metadata_replicas:                        1
  data_replicas:                            1
  metadata_replicas_required:               1
  data_replicas_required:                   1
  encoded_extent_max:                       64.0 KiB
  metadata_checksum:                        none [crc32c] crc64 xxhash 
  data_checksum:                            none [crc32c] crc64 xxhash 
  compression:                              [none] lz4 gzip zstd 
  background_compression:                   [none] lz4 gzip zstd 
  str_hash:                                 crc32c crc64 [siphash] 
  metadata_target:                          none
  foreground_target:                        ssd
  background_target:                        hdd
  promote_target:                           ssd
  erasure_code:                             0
  inodes_32bit:                             1
  shard_inode_numbers:                      1
  inodes_use_key_cache:                     1
  gc_reserve_percent:                       8
  gc_reserve_bytes:                         0 B
  root_reserve_percent:                     0
  wide_macs:                                0
  acl:                                      1
  usrquota:                                 0
  grpquota:                                 0
  prjquota:                                 0
  journal_flush_delay:                      1000
  journal_flush_disabled:                   0
  journal_reclaim_delay:                    100
  journal_transaction_names:                1
  nocow:                                    0

members (size 176):
  Device:                                   0
    UUID:                                   8de13c84-c503-44bc-9ac8-c2732d442708
    Size:                                   954 GiB
    Bucket size:                            512 KiB
    First bucket:                           0
    Buckets:                                1953524
    Last mount:                             Thu Aug  3 19:23:30 2023
    State:                                  rw
    Label:                                  ssd1 (1)
    Data allowed:                           journal,btree,user
    Has data:                               journal,btree,user,cached
    Discard:                                0
    Freespace initialized:                  1
  Device:                                   1
    UUID:                                   2cb0ca1c-d805-4c7c-bbb0-e356bc61e526
    Size:                                   954 GiB
    Bucket size:                            512 KiB
    First bucket:                           0
    Buckets:                                1953524
    Last mount:                             Thu Aug  3 19:23:30 2023
    State:                                  rw
    Label:                                  ssd2 (2)
    Data allowed:                           journal,btree,user
    Has data:                               journal,btree,user,cached
    Discard:                                0
    Freespace initialized:                  1
  Device:                                   2
    UUID:                                   90105af4-ef37-41b0-a77a-01e4a4f7da3e
    Size:                                   14.6 TiB
    Bucket size:                            512 KiB
    First bucket:                           0
    Buckets:                                30519296
    Last mount:                             Thu Aug  3 19:23:30 2023
    State:                                  rw
    Label:                                  hdd1 (4)
    Data allowed:                           journal,btree,user
    Has data:                               user
    Discard:                                0
    Freespace initialized:                  1
bcachefs show-super /dev/sdb
External UUID:                              5c3e0b86-40c3-4bff-8d55-a7de43b9399b
Internal UUID:                              200f8f2d-9ca5-432f-921f-991e42aa19e1
Device index:                               2
Label:                                      
Version:                                    snapshot_trees
Oldest version on disk:                     snapshot_trees
Created:                                    Thu Jul 13 21:41:57 2023
Sequence number:                            40
Superblock size:                            5040
Clean:                                      0
Devices:                                    3
Sections:                                   members,replicas_v0,disk_groups,clean,journal_seq_blacklist,journal_v2,counters
Features:                                   journal_seq_blacklist_v3,new_siphash,inline_data,new_extent_overwrite,btree_ptr_v2,extents_above_btree_updates,btree_updates_journalled,new_varint,journal_no_flush,alloc_v2,extents_across_btree_nodes
Compat features:                            alloc_info,alloc_metadata,extents_above_btree_updates_done,bformat_overflow_done

Options:
  block_size:                               4.00 KiB
  btree_node_size:                          256 KiB
  errors:                                   continue [ro] panic 
  metadata_replicas:                        1
  data_replicas:                            1
  metadata_replicas_required:               1
  data_replicas_required:                   1
  encoded_extent_max:                       64.0 KiB
  metadata_checksum:                        none [crc32c] crc64 xxhash 
  data_checksum:                            none [crc32c] crc64 xxhash 
  compression:                              [none] lz4 gzip zstd 
  background_compression:                   [none] lz4 gzip zstd 
  str_hash:                                 crc32c crc64 [siphash] 
  metadata_target:                          none
  foreground_target:                        ssd
  background_target:                        hddbcachefs show-super /dev/sdb
External UUID:                              5c3e0b86-40c3-4bff-8d55-a7de43b9399b
Internal UUID:                              200f8f2d-9ca5-432f-921f-991e42aa19e1
Device index:                               2
Label:                                      
Version:                                    snapshot_trees
Oldest version on disk:                     snapshot_trees
Created:                                    Thu Jul 13 21:41:57 2023
Sequence number:                            40
Superblock size:                            5040
Clean:                                      0
Devices:                                    3
Sections:                                   members,replicas_v0,disk_groups,clean,journal_seq_blacklist,journal_v2,counters
Features:                                   journal_seq_blacklist_v3,new_siphash,inline_data,new_extent_overwrite,btree_ptr_v2,extents_above_btree_updates,btree_updates_journalled,new_varint,journal_no_flush,alloc_v2,extents_across_btree_nodes
Compat features:                            alloc_info,alloc_metadata,extents_above_btree_updates_done,bformat_overflow_done

Options:
  block_size:                               4.00 KiB
  btree_node_size:                          256 KiB
  errors:                                   continue [ro] panic 
  metadata_replicas:                        1
  data_replicas:                            1
  metadata_replicas_required:               1
  data_replicas_required:                   1
  encoded_extent_max:                       64.0 KiB
  metadata_checksum:                        none [crc32c] crc64 xxhash 
  data_checksum:                            none [crc32c] crc64 xxhash 
  compression:                              [none] lz4 gzip zstd 
  background_compression:                   [none] lz4 gzip zstd 
  str_hash:                                 crc32c crc64 [siphash] 
  metadata_target:                          none
  foreground_target:                        ssd
  background_target:                        hdd
  promote_target:                           ssd
  erasure_code:                             0
  inodes_32bit:                             1
  shard_inode_numbers:                      1
  inodes_use_key_cache:                     1
  gc_reserve_percent:                       8
  gc_reserve_bytes:                         0 B
  root_reserve_percent:                     0
  wide_macs:                                0
  acl:                                      1
  usrquota:                                 0
  grpquota:                                 0
  prjquota:                                 0
  journal_flush_delay:                      1000
  journal_flush_disabled:                   0
  journal_reclaim_delay:                    100
  journal_transaction_names:                1
  nocow:                                    0

members (size 176):
  Device:                                   0
    UUID:                                   8de13c84-c503-44bc-9ac8-c2732d442708
    Size:                                   954 GiB
    Bucket size:                            512 KiB
    First bucket:                           0
    Buckets:                                1953524
    Last mount:                             Thu Aug  3 19:23:30 2023
    State:                                  rw
    Label:                                  ssd1 (1)
    Data allowed:                           journal,btree,user
    Has data:                               journal,btree,user,cached
    Discard:                                0
    Freespace initialized:                  1
  Device:                                   1
    UUID:                                   2cb0ca1c-d805-4c7c-bbb0-e356bc61e526
    Size:                                   954 GiB
    Bucket size:                            512 KiB
    First bucket:                           0
    Buckets:                                1953524
    Last mount:                             Thu Aug  3 19:23:30 2023
    State:                                  rw
    Label:                                  ssd2 (2)
    Data allowed:                           journal,btree,user
    Has data:                               journal,btree,user,cached
    Discard:                                0
    Freespace initialized:                  1
  Device:                                   2
    UUID:                                   90105af4-ef37-41b0-a77a-01e4a4f7da3e
    Size:                                   14.6 TiB
    Bucket size:                            512 KiB
    First bucket:                           0
    Buckets:                                30519296
    Last mount:                             Thu Aug  3 19:23:30 2023
    State:                                  rw
    Label:                                  hdd1 (4)
    Data allowed:                           journal,btree,user
    Has data:                               user
    Discard:                                0
    Freespace initialized:                  1

  promote_target:                           ssd
  erasure_code:                             0
  inodes_32bit:                             1
  shard_inode_numbers:                      1
  inodes_use_key_cache:                     1
  gc_reserve_percent:                       8
  gc_reserve_bytes:                         0 B
  root_reserve_percent:                     0
  wide_macs:                                0
  acl:                                      1
  usrquota:                                 0
  grpquota:                                 0
  prjquota:                                 0
  journal_flush_delay:                      1000
  journal_flush_disabled:                   0
  journal_reclaim_delay:                    100
  journal_transaction_names:                1
  nocow:                                    0

members (size 176):
  Device:                                   0
    UUID:                                   8de13c84-c503-44bc-9ac8-c2732d442708
    Size:                                   954 GiB
    Bucket size:                            512 KiB
    First bucket:                           0
    Buckets:                                1953524
    Last mount:                             Thu Aug  3 19:23:30 2023
    State:                                  rw
    Label:                                  ssd1 (1)
    Data allowed:                           journal,btree,user
    Has data:                               journal,btree,user,cached
    Discard:                                0
    Freespace initialized:                  1
  Device:                                   1
    UUID:                                   2cb0ca1c-d805-4c7c-bbb0-e356bc61e526
    Size:                                   954 GiB
    Bucket size:                            512 KiB
    First bucket:                           0
    Buckets:                                1953524
    Last mount:                             Thu Aug  3 19:23:30 2023
    State:                                  rw
    Label:                                  ssd2 (2)
    Data allowed:                           journal,btree,user
    Has data:                               journal,btree,user,cached
    Discard:                                0
    Freespace initialized:                  1
  Device:                                   2
    UUID:                                   90105af4-ef37-41b0-a77a-01e4a4f7da3e
    Size:                                   14.6 TiB
    Bucket size:                            512 KiB
    First bucket:                           0
    Buckets:                                30519296
    Last mount:                             Thu Aug  3 19:23:30 2023
    State:                                  rw
    Label:                                  hdd1 (4)
    Data allowed:                           journal,btree,user
    Has data:                               user
    Discard:                                0
    Freespace initialized:                  1

Dmesg

[  289.098181] ================================================================================
[  289.098185] UBSAN: shift-out-of-bounds in fs/bcachefs/bkey.c:394:16
[  289.098189] shift exponent 64 is too large for 64-bit type 'long long unsigned int'
[  289.098192] CPU: 12 PID: 215 Comm: kworker/12:1H Tainted: G             L     6.4.0+ #7
[  289.098193] Hardware name: Gigabyte Technology Co., Ltd. B360 HD3P-LM/B360HD3PLM-CF, BIOS F7b HZ 07/29/2021
[  289.098194] Workqueue: bcachefs_io btree_node_read_work
[  289.098198] Call Trace:
[  289.098199]  <TASK>
[  289.098200]  dump_stack_lvl+0x48/0x70
[  289.098203]  dump_stack+0x10/0x20
[  289.098204]  __ubsan_handle_shift_out_of_bounds+0x156/0x310
[  289.098208]  bch2_bkey_pack_pos_lossy.cold+0x8e/0x219
[  289.098211]  __build_ro_aux_tree+0x291/0x820
[  289.098212]  ? __free_pages_ok+0x33a/0x5e0
[  289.098215]  bch2_bset_build_aux_tree+0x13e/0x170
[  289.098216]  bch2_btree_node_read_done+0x10bb/0x1300
[  289.098217]  ? bch2_bkey_ptrs_to_text+0x1a9/0x590
[  289.098219]  ? bch2_btree_ptr_v2_to_text+0x92/0xc0
[  289.098221]  btree_node_read_work+0x28f/0x560
[  289.098222]  ? btree_node_read_work+0x28f/0x560
[  289.098225]  process_one_work+0x21c/0x440
[  289.098227]  worker_thread+0x50/0x3f0
[  289.098228]  ? __pfx_worker_thread+0x10/0x10
[  289.098229]  kthread+0xf4/0x130
[  289.098231]  ? __pfx_kthread+0x10/0x10
[  289.098233]  ret_from_fork+0x29/0x50
[  289.098236]  </TASK>
[  289.098236] ================================================================================
[  296.724378] bcachefs (5c3e0b86-40c3-4bff-8d55-a7de43b9399b): going read-write
[  296.734490] ================================================================================
[  296.734494] UBSAN: shift-out-of-bounds in fs/bcachefs/bkey.c:213:16
[  296.734496] shift exponent 64 is too large for 64-bit type 'long long unsigned int'
[  296.734499] CPU: 12 PID: 3703 Comm: bcachefs Tainted: G             L     6.4.0+ #7
[  296.734500] Hardware name: Gigabyte Technology Co., Ltd. B360 HD3P-LM/B360HD3PLM-CF, BIOS F7b HZ 07/29/2021
[  296.734501] Call Trace:
[  296.734502]  <TASK>
[  296.734504]  dump_stack_lvl+0x48/0x70
[  296.734507]  dump_stack+0x10/0x20
[  296.734509]  __ubsan_handle_shift_out_of_bounds+0x156/0x310
[  296.734513]  ? bch2_bkey_pack_pos_lossy.cold+0x8e/0x219
[  296.734515]  bch2_bkey_pack_key.cold+0x115/0x25e
[  296.734516]  bch2_bset_insert+0x61/0x290
[  296.734519]  ? bch2_btree_path_fix_key_modified+0x1c3/0x1f0
[  296.734521]  bch2_btree_bset_insert_key+0xab/0x4f0
[  296.734524]  ? bch2_journal_keys_peek_slot+0x53/0x80
[  296.734527]  bch2_btree_insert_key_leaf+0x8e/0x2d0
[  296.734528]  __bch2_trans_commit+0x1a90/0x1cd0
[  296.734530]  ? bch2_trans_update+0x7a/0x380
[  296.734532]  bch2_journal_replay+0x335/0x4f0
[  296.734535]  ? bch2_journal_replay+0x25e/0x4f0
[  296.734537]  bch2_fs_recovery+0xbed/0x1d40
[  296.734538]  ? vsnprintf+0x1b9/0x560
[  296.734540]  ? __bch2_sb_field_resize+0x6b/0x140
[  296.734542]  ? __closure_wake_up+0x18/0xf0
[  296.734544]  ? bch2_recalc_capacity+0x246/0x350
[  296.734546]  bch2_fs_start+0x48f/0x4f0
[  296.734547]  bch2_fs_open+0x454/0x5d0
[  296.734549]  bch2_mount+0x557/0x740
[  296.734553]  legacy_get_tree+0x28/0x60
[  296.734554]  vfs_get_tree+0x27/0xe0
[  296.734557]  path_mount+0x4e1/0xbb0
[  296.734559]  ? putname+0x5f/0x80
[  296.734562]  __x64_sys_mount+0x117/0x150
[  296.734563]  do_syscall_64+0x59/0x90
[  296.734566]  ? do_syscall_64+0x69/0x90
[  296.734567]  entry_SYSCALL_64_after_hwframe+0x72/0xdc
[  296.734568] RIP: 0033:0x7f3ec3526eae
[  296.734584] Code: 48 8b 0d 85 1f 0f 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 49 89 ca b8 a5 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 52 1f 0f 00 f7 d8 64 89 01 48
[  296.734585] RSP: 002b:00007ffeb2f243b8 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5
[  296.734587] RAX: ffffffffffffffda RBX: 0000556e0b7780e0 RCX: 00007f3ec3526eae
[  296.734588] RDX: 0000556e0b779da0 RSI: 0000556e0b7780e0 RDI: 0000556e0b779830
[  296.734588] RBP: 0000556e0b779830 R08: 0000000000000000 R09: 8080808080808080
[  296.734589] R10: 0000000000000000 R11: 0000000000000246 R12: 0000556e0b69d958
[  296.734590] R13: 0000000000000023 R14: 0000556e0b779da0 R15: 0000000000000006
[  296.734591]  </TASK>
[  296.734594] ================================================================================
[  339.153332] bcachefs (5c3e0b86-40c3-4bff-8d55-a7de43b9399b): error validating btree node on nvme0n1 at btree extents level 0/2
[  339.153334]   u64s 11 type btree_ptr_v2 1744968794:174456:U32_MAX len 0 ver 0: seq ccefcb08f8a2d0d5 written 344 min_key 1744968793:2975568841:0 durability: 1 ptr: 0:233814:0 gen 23
[  339.153335]   node offset 0 bset u64s 12343: invalid bkey format: field too large
[  340.181882] bcachefs (5c3e0b86-40c3-4bff-8d55-a7de43b9399b): corrupt btree node before write at btree extents level 0/2
[  340.181883]   u64s 11 type btree_ptr_v2 1744968794:174456:U32_MAX len 0 ver 0: seq 8b5c1a6367725481 written 320 min_key 1744968793:2975568841:0 durability: 1 ptr: 0:1363790:512 gen 10
[  340.181884]   node offset 0 bset u64s 20023: invalid bkey format: field too large
[  353.449159] bcachefs (5c3e0b86-40c3-4bff-8d55-a7de43b9399b): journal replay done
[  354.248683] bcachefs (5c3e0b86-40c3-4bff-8d55-a7de43b9399b): mounted version 29: snapshot_trees opts=foreground_target=ssd,background_target=hdd,promote_target=ssd
[ 1758.336320] bcachefs (5c3e0b86-40c3-4bff-8d55-a7de43b9399b): recovering from clean shutdown, journal seq 11435210
[ 1758.386095] bcachefs (5c3e0b86-40c3-4bff-8d55-a7de43b9399b): going read-write
[ 1758.403370] bcachefs (5c3e0b86-40c3-4bff-8d55-a7de43b9399b): mounted version 29: snapshot_trees opts=foreground_target=ssd,background_target=hdd,promote_target=ssd
[ 2685.773850] bcachefs (5c3e0b86-40c3-4bff-8d55-a7de43b9399b): recovering from clean shutdown, journal seq 11437512
[ 2685.823866] bcachefs (5c3e0b86-40c3-4bff-8d55-a7de43b9399b): going read-write
[ 2685.838967] bcachefs (5c3e0b86-40c3-4bff-8d55-a7de43b9399b): mounted version 29: snapshot_trees opts=foreground_target=ssd,background_target=hdd,promote_target=ssd
Doc-Pixel commented 1 year ago

I can mount readonly mount -t bcachefs /dev/nvme0n1:/dev/nvme1n1:/dev/sdb /data -o ro,nochanges,norecovery and see the data.

koverstreet commented 1 year ago

Are you on IRC?

Since you can mount readonly, the dump tool should work - can you send me your filesystem metadata?

Doc-Pixel commented 1 year ago

I'm in the IRC now, will pass a link to the files.

koverstreet commented 1 year ago

Closing since we were unable to repro, reopen if needed