search

koverstreet / bcachefs

Other
643 stars 71 forks source link

Filesystem hangs after going read-write #648

Open Snogard opened 5 months ago

Snogard commented 5 months ago

OS: Archlinux Kernel: 6.7.3-zen1-1-zen

Context

I have two 8tb disks in the same pool with replicas=2, one of the disks needed to be replaced so i tried doing these steps:

bcachefs device add /dev/disk/by-id/new-disk
bcachefs device evacuate /dev/disk/by-id/old-disk

problem is, the process hanged and i had to restart my pc. Next time, i tried mounting with fsck,fix_errors because some journal entries went missing, it seems to go fine until it goes read-write and hangs again.

After another system restart, the errors are still the same and i can't seem to mount the filesystem in any way.
i know bcachefs is still in beta, but i woud really like to recover my data if possible, thanks in advance for the help.

Originally i formatted the filesystem with this command:

bcachefs format \
    --background_compression=zstd \
    --replicas=2 \
    --label=hdd.hdd1 /dev/disk/by-id/disk1 \
    --label=hdd.hdd2 /dev/disk/by-id/disk2 \
    --background_target=hdd \
    --metadata_target=hdd

Logs

bcachefs super:

External UUID:                              d6020e9b-770a-4aa5-a4af-389f4003b650
Internal UUID:                              abd2fad9-022a-4c22-af54-4b8d764050c1
Magic number:                               c68573f6-66ce-90a9-d96a-60cf803df7ef
Device index:                               0
Label:                                      
Version:                                    1.3: rebalance_work
Version upgrade complete:                   1.3: rebalance_work
Oldest version on disk:                     0.29: snapshot_trees
Created:                                    Fri Jun 30 14:07:06 2023
Sequence number:                            864
Time of last write:                         Thu Jan  1 01:00:00 1970
Superblock size:                            5760
Clean:                                      0
Devices:                                    3
Sections:                                   members_v1,replicas_v0,disk_groups,clean,journal_seq_blacklist,journal_v2,counters,members_v2,errors,ext,downgrade
Features:                                   zstd,journal_seq_blacklist_v3,reflink,new_siphash,inline_data,new_extent_overwrite,btree_ptr_v2,extents_above_btree_updates,btree_updates_journalled,reflink_inline_data,new_varint,journal_no_flush,alloc_v2,extents_across_btree_nodes
Compat features:                            alloc_info,alloc_metadata,extents_above_btree_updates_done,bformat_overflow_done

Options:
  block_size:                               4.00 KiB
  btree_node_size:                          256 KiB
  errors:                                   continue [ro] panic 
  metadata_replicas:                        2
  data_replicas:                            2
  metadata_replicas_required:               1
  data_replicas_required:                   1
  encoded_extent_max:                       64.0 KiB
  metadata_checksum:                        none [crc32c] crc64 xxhash 
  data_checksum:                            none [crc32c] crc64 xxhash 
  compression:                              none
  background_compression:                   zstd
  str_hash:                                 crc32c crc64 [siphash] 
  metadata_target:                          hdd
  foreground_target:                        none
  background_target:                        hdd
  promote_target:                           none
  erasure_code:                             0
  inodes_32bit:                             1
  shard_inode_numbers:                      1
  inodes_use_key_cache:                     1
  gc_reserve_percent:                       8
  gc_reserve_bytes:                         0 B
  root_reserve_percent:                     0
  wide_macs:                                0
  acl:                                      1
  usrquota:                                 0
  grpquota:                                 0
  prjquota:                                 0
  journal_flush_delay:                      1000
  journal_flush_disabled:                   0
  journal_reclaim_delay:                    100
  journal_transaction_names:                1
  version_upgrade:                          [compatible] incompatible none 
  nocow:                                    0

members_v2 (size 376):
Device:                                     0
  Label:                                    hdd1 (1)
  UUID:                                     3042d1a2-1607-48b2-a39b-8ea110cb90eb
  Size:                                     7.28 TiB
  read errors:                              0
  write errors:                             0
  checksum errors:                          0
  seqread iops:                             0
  seqwrite iops:                            0
  randread iops:                            0
  randwrite iops:                           0
  Bucket size:                              512 KiB
  First bucket:                             0
  Buckets:                                  15261770
  Last mount:                               Sat Feb  3 20:22:20 2024
  Last superblock write:                    0
  State:                                    rw
  Data allowed:                             journal,btree,user
  Has data:                                 journal,btree,user,cached
  Durability:                               1
  Discard:                                  0
  Freespace initialized:                    1
Device:                                     1
  Label:                                    hdd2 (2)
  UUID:                                     1788f537-3cd4-46c7-91c3-b5a91be60be4
  Size:                                     7.28 TiB
  read errors:                              26
  write errors:                             0
  checksum errors:                          0
  seqread iops:                             0
  seqwrite iops:                            0
  randread iops:                            0
  randwrite iops:                           0
  Bucket size:                              512 KiB
  First bucket:                             0
  Buckets:                                  15261770
  Last mount:                               Sat Feb  3 20:22:20 2024
  Last superblock write:                    0
  State:                                    failed
  Data allowed:                             journal,btree,user
  Has data:                                 journal,btree,user,cached
  Durability:                               1
  Discard:                                  0
  Freespace initialized:                    1
Device:                                     2
  Label:                                    hdd3 (3)
  UUID:                                     0e8d4550-ce6f-4436-b4eb-71ee38c901b8
  Size:                                     7.28 TiB
  read errors:                              0
  write errors:                             0
  checksum errors:                          0
  seqread iops:                             0
  seqwrite iops:                            0
  randread iops:                            0
  randwrite iops:                           0
  Bucket size:                              1.00 MiB
  First bucket:                             0
  Buckets:                                  7630885
  Last mount:                               Sat Feb  3 20:22:20 2024
  Last superblock write:                    0
  State:                                    rw
  Data allowed:                             journal,btree,user
  Has data:                                 journal,btree,user
  Durability:                               1
  Discard:                                  0
  Freespace initialized:                    1

errors (size 152):
journal_entries_missing                     314             Sat Feb  3 20:22:54 2024
journal_entry_replicas_not_marked           1               Wed Dec 13 14:59:42 2023
fs_usage_hidden_wrong                       1               Sat Feb  3 18:18:18 2024
dev_usage_buckets_wrong                     3               Sat Feb  3 18:18:18 2024
dev_usage_sectors_wrong                     2               Sat Feb  3 18:18:18 2024
dev_usage_fragmented_wrong                  1               Sat Feb  3 18:18:18 2024
alloc_key_data_type_wrong                   16398           Sat Feb  3 19:33:22 2024
alloc_key_dirty_sectors_wrong               16398           Sat Feb  3 19:33:22 2024
extent_in_missing_inode                     25              Thu Dec 14 03:40:16 2023

kernel log:

[Feb 3 20:22] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): mounting version 1.3: rebalance_work opts=metadata_replicas=2,data_replicas=2,compression=zstd,metadata_target=hdd,background_target=hdd,degraded,verbose,fsck,fix_errors=yes
[  +0.000005] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): recovering from unclean shutdown
[  +0.000002] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): starting journal read
[ +17.465563] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): journal read done on device sdf, ret 0
[ +17.167794] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): journal read done on device sda, ret 0
[  +0.000012] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): journal read done, replaying entries 45201674-45202612
[  +0.000090] journal entries 45202101-45202108 missing! (replaying 45201674-45202612)
[  +0.000000]   prev at 0:1041:432 (sector 1071536) 2:606:992 (sector 1248224) size 8
[  +0.000001]   next at 0:1041:440 (sector 1071544) 2:606:1000 (sector 1248232), fixing
[  +0.000005] journal entries 45202143-45202150 missing! (replaying 45201674-45202612)
[  +0.000000]   prev at 0:1043:240 (sector 1073392) 2:607:816 (sector 1250096) size 16
[  +0.000001]   next at 0:1043:256 (sector 1073408) 2:607:832 (sector 1250112), fixing
[  +0.000004] journal entries 45202196-45202203 missing! (replaying 45201674-45202612)
[  +0.000001]   prev at 0:1045:344 (sector 1075544) 2:608:904 (sector 1252232) size 16
[  +0.000000]   next at 0:1045:360 (sector 1075560) 2:608:920 (sector 1252248), fixing
[  +0.000004] journal entries 45202254-45202261 missing! (replaying 45201674-45202612)
[  +0.000001]   prev at 0:1047:496 (sector 1077744) 2:609:1096 (sector 1254472) size 8
[  +0.000000]   next at 0:1047:504 (sector 1077752) 2:609:1104 (sector 1254480), fixing
[  +0.000004] journal entries 45202300-45202307 missing! (replaying 45201674-45202612)
[  +0.000001]   prev at 0:1049:344 (sector 1079640) 2:610:960 (sector 1256384) size 8
[  +0.000000]   next at 0:1049:352 (sector 1079648) 2:610:968 (sector 1256392), fixing
[  +0.000004] journal entries 45202358-45202365 missing! (replaying 45201674-45202612)
[  +0.000001]   prev at 0:1051:992 (sector 1082336) 2:611:1576 (sector 1259048) size 16
[  +0.000000]   next at 0:1052:0 (sector 1082368) 2:611:1592 (sector 1259064), fixing
[  +0.470607] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): Journal keys: 924354 read, 528450 after sorting and compacting
[  +0.108651] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): alloc_read... done
[  +0.888153] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): stripes_read... done
[  +0.000005] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): snapshots_read... done
[  +0.000025] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): check_allocations...
[Feb 3 20:49] zsh (19218): drop_caches: 3
[Feb 3 21:09] zsh (19218): drop_caches: 3
[Feb 3 21:10] dev 1 has wrong sb buckets: got 0, should be 7, fixing
[  +0.000005] dev 1 has wrong sb sectors: got 0, should be 6152, fixing
[  +0.000003] dev 1 has wrong sb fragmented: got 0, should be 1016, fixing
[  +0.000002] dev 1 has wrong journal buckets: got 0, should be 8192, fixing
[  +0.000002] dev 1 has wrong journal sectors: got 0, should be 8388608, fixing
[  +0.000003] dev 1 has wrong need_discard buckets: got 8588, should be 389, fixing
[  +0.000003] fs has wrong hidden: got 25181184, should be 33576960, fixing
[  +0.012255]  done
[  +0.182778] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): journal_replay...
[  +0.132765] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): going read-write
[Feb 3 21:14] INFO: task kworker/u32:0:6605 blocked for more than 122 seconds.
[  +0.000004]       Tainted: P           OE      6.7.3-zen1-1-zen #1
[  +0.000002] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  +0.000001] task:kworker/u32:0   state:D stack:0     pid:6605  tgid:6605  ppid:2      flags:0x00004000
[  +0.000005] Workqueue: btree_update btree_interior_update_work [bcachefs]
[  +0.000041] Call Trace:
[  +0.000001]  <TASK>
[  +0.000002]  __schedule+0xcaa/0x1950
[  +0.000005]  ? srso_alias_return_thunk+0x5/0xfbef5
[  +0.000004]  ? __bch2_time_stats_update+0x11c/0x270 [bcachefs d16c7c3dd27d4670c6d6b654661931afac1c615e]
[  +0.000043]  schedule+0x32/0xd0
[  +0.000002]  __closure_sync+0x82/0x160
[  +0.000003]  bch2_btree_update_start+0x91a/0x940 [bcachefs d16c7c3dd27d4670c6d6b654661931afac1c615e]
[  +0.000039]  ? __pfx_closure_sync_fn+0x10/0x10
[  +0.000003]  ? bch2_btree_split_leaf+0x46/0x310 [bcachefs d16c7c3dd27d4670c6d6b654661931afac1c615e]
[  +0.000034]  bch2_btree_split_leaf+0x46/0x310 [bcachefs d16c7c3dd27d4670c6d6b654661931afac1c615e]
[  +0.000034]  ? btree_interior_update_work+0x983/0xae0 [bcachefs d16c7c3dd27d4670c6d6b654661931afac1c615e]
[  +0.000033]  bch2_trans_commit_error+0x6c/0x640 [bcachefs d16c7c3dd27d4670c6d6b654661931afac1c615e]
[  +0.000036]  __bch2_trans_commit+0xd42/0x2050 [bcachefs d16c7c3dd27d4670c6d6b654661931afac1c615e]
[  +0.000036]  btree_interior_update_work+0x983/0xae0 [bcachefs d16c7c3dd27d4670c6d6b654661931afac1c615e]
[  +0.000035]  process_one_work+0x171/0x330
[  +0.000003]  worker_thread+0x3ef/0x580
[  +0.000003]  ? __pfx_worker_thread+0x10/0x10
[  +0.000001]  kthread+0xe5/0x120
[  +0.000003]  ? __pfx_kthread+0x10/0x10
[  +0.000002]  ret_from_fork+0x31/0x50
[  +0.000003]  ? __pfx_kthread+0x10/0x10
[  +0.000002]  ret_from_fork_asm+0x1b/0x30
[  +0.000005]  </TASK>
[  +0.000035] INFO: task bcachefs:11176 blocked for more than 122 seconds.
[  +0.000001]       Tainted: P           OE      6.7.3-zen1-1-zen #1
[  +0.000002] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  +0.000000] task:bcachefs        state:D stack:0     pid:11176 tgid:11176 ppid:11175  flags:0x00004002
[  +0.000003] Call Trace:
[  +0.000001]  <TASK>
[  +0.000002]  __schedule+0xcaa/0x1950
[  +0.000003]  ? srso_alias_return_thunk+0x5/0xfbef5
[  +0.000002]  ? __bch2_time_stats_update+0x11c/0x270 [bcachefs d16c7c3dd27d4670c6d6b654661931afac1c615e]
[  +0.000038]  schedule+0x32/0xd0
[  +0.000002]  __closure_sync+0x82/0x160
[  +0.000002]  bch2_btree_update_start+0x91a/0x940 [bcachefs d16c7c3dd27d4670c6d6b654661931afac1c615e]
[  +0.000034]  ? srso_alias_return_thunk+0x5/0xfbef5
[  +0.000004]  ? __pfx_closure_sync_fn+0x10/0x10
[  +0.000003]  ? bch2_btree_split_leaf+0x46/0x310 [bcachefs d16c7c3dd27d4670c6d6b654661931afac1c615e]
[  +0.000033]  bch2_btree_split_leaf+0x46/0x310 [bcachefs d16c7c3dd27d4670c6d6b654661931afac1c615e]
[  +0.000033]  ? bch2_journal_replay+0x2fc/0x5d0 [bcachefs d16c7c3dd27d4670c6d6b654661931afac1c615e]
[  +0.000040]  bch2_trans_commit_error+0x6c/0x640 [bcachefs d16c7c3dd27d4670c6d6b654661931afac1c615e]
[  +0.000036]  __bch2_trans_commit+0xd42/0x2050 [bcachefs d16c7c3dd27d4670c6d6b654661931afac1c615e]
[  +0.000036]  bch2_journal_replay+0x2fc/0x5d0 [bcachefs d16c7c3dd27d4670c6d6b654661931afac1c615e]
[  +0.000039]  bch2_fs_recovery+0x18ab/0x1be0 [bcachefs d16c7c3dd27d4670c6d6b654661931afac1c615e]
[  +0.000036]  ? srso_alias_return_thunk+0x5/0xfbef5
[  +0.000002]  ? bch2_printbuf_exit+0x20/0x30 [bcachefs d16c7c3dd27d4670c6d6b654661931afac1c615e]
[  +0.000035]  ? srso_alias_return_thunk+0x5/0xfbef5
[  +0.000002]  ? print_mount_opts+0x4b6/0x630 [bcachefs d16c7c3dd27d4670c6d6b654661931afac1c615e]
[  +0.000039]  ? srso_alias_return_thunk+0x5/0xfbef5
[  +0.000003]  bch2_fs_start+0x32f/0x3b0 [bcachefs d16c7c3dd27d4670c6d6b654661931afac1c615e]
[  +0.000034]  bch2_fs_open+0x1141/0x1880 [bcachefs d16c7c3dd27d4670c6d6b654661931afac1c615e]
[  +0.000029]  ? submit_bio_noacct+0x4c/0x590
[  +0.000007]  ? srso_alias_return_thunk+0x5/0xfbef5
[  +0.000002]  ? bch2_mount+0x4e5/0x720 [bcachefs d16c7c3dd27d4670c6d6b654661931afac1c615e]
[  +0.000028]  bch2_mount+0x4e5/0x720 [bcachefs d16c7c3dd27d4670c6d6b654661931afac1c615e]
[  +0.000028]  legacy_get_tree+0x28/0x50
[  +0.000003]  vfs_get_tree+0x26/0xf0
[  +0.000002]  ? srso_alias_return_thunk+0x5/0xfbef5
[  +0.000001]  path_mount+0x4c9/0xb80
[  +0.000003]  __x64_sys_mount+0x11a/0x150
[  +0.000002]  do_syscall_64+0x61/0xe0
[  +0.000002]  ? srso_alias_return_thunk+0x5/0xfbef5
[  +0.000001]  ? __x64_sys_write+0xdd/0xf0
[  +0.000002]  ? srso_alias_return_thunk+0x5/0xfbef5
[  +0.000001]  ? syscall_exit_to_user_mode+0x2b/0x40
[  +0.000002]  ? srso_alias_return_thunk+0x5/0xfbef5
[  +0.000001]  ? do_syscall_64+0x70/0xe0
[  +0.000001]  entry_SYSCALL_64_after_hwframe+0x6e/0x76
[  +0.000002] RIP: 0033:0x71fb953caafe
[  +0.000022] RSP: 002b:00007ffeaaacf448 EFLAGS: 00000206 ORIG_RAX: 00000000000000a5
[  +0.000002] RAX: ffffffffffffffda RBX: 00005f91588452d0 RCX: 000071fb953caafe
[  +0.000001] RDX: 00005f91588452d0 RSI: 00005f9158843e60 RDI: 00005f91588479d0
[  +0.000001] RBP: 00005f9158847480 R08: 00005f9158847480 R09: 0000000000000007
[  +0.000000] R10: 0000000002200000 R11: 0000000000000206 R12: 00005f91588479d0
[  +0.000001] R13: 0000000000000090 R14: 00005f9158848200 R15: 0000000000000004
[  +0.000003]  </TASK>
koverstreet commented 5 months ago

That looks like it's stuck trying to allocate - are your disks all full?

You can check dev-*/alloc_debug in sysfs - while it's trying to mount bcachefs fs usage won't work, but sysfs is accessible

Snogard commented 5 months ago

before starting the whole process i remember about 850gb being free out of 13tb (reported by dolphin), so about half per disk ideally. The last disk i added (hdd3) was empty and never got to be fully used since evacaute hanged and i never got the chance to launch a rereplicate job.

Anyway, here are the logs:

/sys/fs/bcachefs/d6020e9b-770a-4aa5-a4af-389f4003b650/dev-0/alloc_debug:

                 buckets         sectors      fragmented
free             1895418               0               0
sb                     7            6152            1016
journal             8192         8388608               0
btree             186693       111556608        79617024
user            13145427     13321947122       138970126
cached             26033        13916696               0
parity                 0               0               0
stripe                 0               0               0
need_gc_gens           0               0               0
need_discard           0               0               0
ec                     0

reserves:
stripe            476958
normal            238493
copygc                28
btree                 14
btree_copygc           0
reclaim                0

freelist_wait           empty
open buckets allocated  1
open buckets this dev   0
open buckets total      1024
open_buckets_wait       empty
open_buckets_btree      0
open_buckets_user       0
buckets_to_invalidate   0
btree reserve cache     0

/sys/fs/bcachefs/d6020e9b-770a-4aa5-a4af-389f4003b650/dev-1/alloc_debug:

                 buckets         sectors      fragmented
free             1896865               0               0
sb                     7            6152            1016
journal             8192         8388608               0
btree             185251       110264832        79432192
user            13145428     13321947546       138970726
cached             26027        13927360               0
parity                 0               0               0
stripe                 0               0               0
need_gc_gens           0               0               0
need_discard           0               0               0
ec                     0

reserves:
stripe            476958
normal            238493
copygc                28
btree                 14
btree_copygc           0
reclaim                0

freelist_wait           empty
open buckets allocated  1
open buckets this dev   0
open buckets total      1024
open_buckets_wait       empty
open_buckets_btree      0
open_buckets_user       0
buckets_to_invalidate   0
btree reserve cache     0

/sys/fs/bcachefs/d6020e9b-770a-4aa5-a4af-389f4003b650/dev-2/alloc_debug:

                 buckets         sectors      fragmented
free             7608453               0               0
sb                     4            6152            2040
journal             8192        16777216               0
btree               1110         1783296          489984
user               13126        26880424            1624
cached                 0               0               0
parity                 0               0               0
stripe                 0               0               0
need_gc_gens           0               0               0
need_discard           0               0               0
ec                     0

reserves:
stripe            238478
normal            119246
copygc                14
btree                  7
btree_copygc           0
reclaim                0

freelist_wait           empty
open buckets allocated  1
open buckets this dev   0
open buckets total      1024
open_buckets_wait       empty
open_buckets_btree      0
open_buckets_user       0
buckets_to_invalidate   0
btree reserve cache     0
koverstreet commented 5 months ago

Can you faddr2line the dmesg output?

Snogard commented 5 months ago

It's my first time doing something like this and after a lot of hours i think i need some guidance on this.

I did recompile the kernel without stripping, i used the arch linux's pkgbuild, i commented out the "stripping build tools" lines and "stripping vm linux" lines.

After a reboot i tried to mount the pool to get a fresh dmesg from the new kernel and i used that for faddr2line: /usr/lib/modules/6.7.3-zen1-2-zen/build/scripts/faddr2line /usr/lib/modules/6.7.3-zen1-2-zen/build/vmlinux ....

The only problem is that i can't seem to get a match for the bcachefs stuff, i did try to use faddress on the uncompressed bcachefs module found at /usr/lib/modules/6.7.3-zen1-2-zen/kernel/fs/bcachefs/bcachefs.ko.zst but it still gives me this error: ERROR: CONFIG_DEBUG_INFO not enabled I feel a bit lost.

file log for bcachefs.ko: bcachefs.ko: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), BuildID[sha1]=bc7df9d851182d3b23b2da70a497331d07bf91c0, not stripped

regular dmesg ``` [ +0.316654] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): mounting version 1.3: rebalance_work opts=metadata_replicas=2,data_replicas=2,compression=zstd,metadata_target=hdd,background_target=hdd,degraded,verbose,fsck,fix_errors=yes [ +0.000004] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): recovering from unclean shutdown [ +0.000002] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): starting journal read [Feb 5 14:34] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): journal read done on device sdc, ret 0 [ +17.183197] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): journal read done on device sdb, ret 0 [ +0.000029] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): journal read done, replaying entries 45201684-45202738 [ +0.000127] journal entries 45202101-45202108 missing! (replaying 45201684-45202738) [ +0.000002] prev at 0:1041:432 (sector 1071536) 2:606:992 (sector 1248224) size 8 [ +0.000000] next at 0:1041:440 (sector 1071544) 2:606:1000 (sector 1248232), fixing [ +0.000008] journal entries 45202143-45202150 missing! (replaying 45201684-45202738) [ +0.000001] prev at 0:1043:240 (sector 1073392) 2:607:816 (sector 1250096) size 16 [ +0.000000] next at 0:1043:256 (sector 1073408) 2:607:832 (sector 1250112), fixing [ +0.000007] journal entries 45202196-45202203 missing! (replaying 45201684-45202738) [ +0.000000] prev at 0:1045:344 (sector 1075544) 2:608:904 (sector 1252232) size 16 [ +0.000001] next at 0:1045:360 (sector 1075560) 2:608:920 (sector 1252248), fixing [ +0.000006] journal entries 45202254-45202261 missing! (replaying 45201684-45202738) [ +0.000001] prev at 0:1047:496 (sector 1077744) 2:609:1096 (sector 1254472) size 8 [ +0.000000] next at 0:1047:504 (sector 1077752) 2:609:1104 (sector 1254480), fixing [ +0.000006] journal entries 45202300-45202307 missing! (replaying 45201684-45202738) [ +0.000001] prev at 0:1049:344 (sector 1079640) 2:610:960 (sector 1256384) size 8 [ +0.000000] next at 0:1049:352 (sector 1079648) 2:610:968 (sector 1256392), fixing [ +0.000006] journal entries 45202358-45202365 missing! (replaying 45201684-45202738) [ +0.000001] prev at 0:1051:992 (sector 1082336) 2:611:1576 (sector 1259048) size 16 [ +0.000001] next at 0:1052:0 (sector 1082368) 2:611:1592 (sector 1259064), fixing [ +0.507274] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): Journal keys: 913957 read, 522621 after sorting and compacting [ +0.288440] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): alloc_read... done [ +1.365527] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): stripes_read... done [ +0.000005] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): snapshots_read... done [ +0.000025] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): check_allocations... done [Feb 5 15:26] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): journal_replay... [ +0.135301] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): going read-write [Feb 5 15:31] INFO: task kworker/u16:1:13 blocked for more than 122 seconds. [ +0.000005] Not tainted 6.7.3-zen1-2-zen #1 [ +0.000002] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ +0.000001] task:kworker/u16:1 state:D stack:0 pid:13 tgid:13 ppid:2 flags:0x00004000 [ +0.000004] Workqueue: btree_update btree_interior_update_work [bcachefs] [ +0.000043] Call Trace: [ +0.000001] [ +0.000003] __schedule+0xcaa/0x1950 [ +0.000006] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000003] ? __bch2_time_stats_update+0x11c/0x270 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000040] schedule+0x32/0xd0 [ +0.000003] __closure_sync+0x82/0x160 [ +0.000003] bch2_btree_update_start+0x917/0x940 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000047] ? __pfx_closure_sync_fn+0x10/0x10 [ +0.000004] ? bch2_btree_split_leaf+0x46/0x310 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000030] bch2_btree_split_leaf+0x46/0x310 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000030] ? btree_interior_update_work+0x98d/0xaf0 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000027] bch2_trans_commit_error+0x6c/0x640 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000022] __bch2_trans_commit+0xd42/0x2050 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000022] btree_interior_update_work+0x98d/0xaf0 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000021] process_one_work+0x178/0x340 [ +0.000002] worker_thread+0x301/0x490 [ +0.000002] ? __pfx_worker_thread+0x10/0x10 [ +0.000002] kthread+0xe5/0x120 [ +0.000001] ? __pfx_kthread+0x10/0x10 [ +0.000002] ret_from_fork+0x31/0x50 [ +0.000002] ? __pfx_kthread+0x10/0x10 [ +0.000002] ret_from_fork_asm+0x1b/0x30 [ +0.000004] [ +0.000040] INFO: task bcachefs:1360 blocked for more than 122 seconds. [ +0.000001] Not tainted 6.7.3-zen1-2-zen #1 [ +0.000001] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ +0.000000] task:bcachefs state:D stack:0 pid:1360 tgid:1360 ppid:1359 flags:0x00004002 [ +0.000002] Call Trace: [ +0.000001] [ +0.000001] __schedule+0xcaa/0x1950 [ +0.000002] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000002] ? __bch2_time_stats_update+0x11c/0x270 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000023] schedule+0x32/0xd0 [ +0.000002] __closure_sync+0x82/0x160 [ +0.000001] bch2_btree_update_start+0x917/0x940 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000021] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000001] ? bch2_btree_node_iter_init+0x1d2/0x8b0 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000022] ? __pfx_closure_sync_fn+0x10/0x10 [ +0.000002] ? bch2_btree_split_leaf+0x46/0x310 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000020] bch2_btree_split_leaf+0x46/0x310 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000019] ? bch2_journal_replay+0x2fc/0x5d0 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000026] bch2_trans_commit_error+0x6c/0x640 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000021] __bch2_trans_commit+0xd42/0x2050 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000022] bch2_journal_replay+0x2fc/0x5d0 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000023] bch2_fs_recovery+0x18ab/0x1be0 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000021] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000001] ? bch2_printbuf_exit+0x20/0x30 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000022] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000001] ? print_mount_opts+0x4b6/0x630 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000024] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000002] bch2_fs_start+0x32f/0x3b0 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000021] bch2_fs_open+0x1158/0x18d0 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000025] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000003] ? bch2_mount+0x4e5/0x720 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000025] bch2_mount+0x4e5/0x720 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000024] legacy_get_tree+0x28/0x50 [ +0.000002] vfs_get_tree+0x26/0xf0 [ +0.000002] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000002] path_mount+0x4c9/0xb80 [ +0.000002] __x64_sys_mount+0x11a/0x150 [ +0.000002] do_syscall_64+0x61/0xe0 [ +0.000003] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000001] ? __x64_sys_write+0xdd/0xf0 [ +0.000002] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000002] ? syscall_exit_to_user_mode+0x2b/0x40 [ +0.000001] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000002] ? do_syscall_64+0x70/0xe0 [ +0.000001] ? do_syscall_64+0x70/0xe0 [ +0.000002] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000001] ? exc_page_fault+0x7f/0x180 [ +0.000002] entry_SYSCALL_64_after_hwframe+0x6e/0x76 [ +0.000002] RIP: 0033:0x7f1f9f267d2e [ +0.000012] RSP: 002b:00007ffda0b30cb8 EFLAGS: 00000206 ORIG_RAX: 00000000000000a5 [ +0.000002] RAX: ffffffffffffffda RBX: 00005eee92470d80 RCX: 00007f1f9f267d2e [ +0.000001] RDX: 00005eee9246e590 RSI: 00005eee9246ea00 RDI: 00005eee92470e80 [ +0.000001] RBP: 00005eee92470d80 R08: 00005eee92470d80 R09: 0000000000000007 [ +0.000000] R10: 0000000002200000 R11: 0000000000000206 R12: 00005eee92470e80 [ +0.000001] R13: 0000000000000090 R14: 00005eee924716b0 R15: 0000000000000004 [ +0.000003] ```
addressed dmesg ``` [Feb 5 15:31] INFO: task kworker/u16:1:13 blocked for more than 122 seconds. [ +0.000005] Not tainted 6.7.3-zen1-2-zen #1 [ +0.000002] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ +0.000001] task:kworker/u16:1 state:D stack:0 pid:13 tgid:13 ppid:2 flags:0x00004000 [ +0.000004] Workqueue: btree_update btree_interior_update_work [bcachefs] [ +0.000043] Call Trace: [ +0.000001] [ +0.000003] context_switch at kernel/sched/core.c:5376 (inlined by) __schedule at kernel/sched/core.c:6688 [ +0.000006] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:181 [ +0.000003] ? __bch2_time_stats_update+0x11c/0x270 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000040] __preempt_count_sub at arch/x86/include/asm/preempt.h:85 (discriminator 13) (inlined by) __schedule_loop at kernel/sched/core.c:6764 (discriminator 13) (inlined by) schedule at kernel/sched/core.c:6778 (discriminator 13) [ +0.000003] __closure_sync at lib/closure.c:132 (discriminator 4) [ +0.000003] bch2_btree_update_start+0x917/0x940 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000047] ? closure_sync_fn at lib/closure.c:112 [ +0.000004] ? bch2_btree_split_leaf+0x46/0x310 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000030] bch2_btree_split_leaf+0x46/0x310 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000030] ? btree_interior_update_work+0x98d/0xaf0 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000027] bch2_trans_commit_error+0x6c/0x640 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000022] __bch2_trans_commit+0xd42/0x2050 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000022] btree_interior_update_work+0x98d/0xaf0 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000021] process_one_work at kernel/workqueue.c:2642 [ +0.000002] process_scheduled_works at kernel/workqueue.c:2709 (discriminator 2) (inlined by) worker_thread at kernel/workqueue.c:2796 (discriminator 2) [ +0.000002] ? worker_thread at kernel/workqueue.c:2742 [ +0.000002] kthread at kernel/kthread.c:388 [ +0.000001] ? kthread at kernel/kthread.c:341 [ +0.000002] ret_from_fork at arch/x86/kernel/process.c:147 [ +0.000002] ? kthread at kernel/kthread.c:341 [ +0.000002] ret_from_fork_asm at arch/x86/entry/entry_64.S:250 [ +0.000004] [ +0.000040] INFO: task bcachefs:1360 blocked for more than 122 seconds. [ +0.000001] Not tainted 6.7.3-zen1-2-zen #1 [ +0.000001] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ +0.000000] task:bcachefs state:D stack:0 pid:1360 tgid:1360 ppid:1359 flags:0x00004002 [ +0.000002] Call Trace: [ +0.000001] [ +0.000001] context_switch at kernel/sched/core.c:5376 (inlined by) __schedule at kernel/sched/core.c:6688 [ +0.000002] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:181 [ +0.000002] ? __bch2_time_stats_update+0x11c/0x270 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000023] __preempt_count_sub at arch/x86/include/asm/preempt.h:85 (discriminator 13) (inlined by) __schedule_loop at kernel/sched/core.c:6764 (discriminator 13) (inlined by) schedule at kernel/sched/core.c:6778 (discriminator 13) [ +0.000002] __closure_sync at lib/closure.c:132 (discriminator 4) [ +0.000001] bch2_btree_update_start+0x917/0x940 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000021] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:181 [ +0.000001] ? bch2_btree_node_iter_init+0x1d2/0x8b0 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000022] ? closure_sync_fn at lib/closure.c:112 [ +0.000002] ? bch2_btree_split_leaf+0x46/0x310 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000020] bch2_btree_split_leaf+0x46/0x310 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000019] ? bch2_journal_replay+0x2fc/0x5d0 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000026] bch2_trans_commit_error+0x6c/0x640 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000021] __bch2_trans_commit+0xd42/0x2050 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000022] bch2_journal_replay+0x2fc/0x5d0 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000023] bch2_fs_recovery+0x18ab/0x1be0 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000021] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:181 [ +0.000001] ? bch2_printbuf_exit+0x20/0x30 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000022] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:181 [ +0.000001] ? print_mount_opts+0x4b6/0x630 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000024] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:181 [ +0.000002] bch2_fs_start+0x32f/0x3b0 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000021] bch2_fs_open+0x1158/0x18d0 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000025] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:181 [ +0.000003] ? bch2_mount+0x4e5/0x720 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000025] bch2_mount+0x4e5/0x720 [bcachefs bc7df9d851182d3b23b2da70a497331d07bf91c0] [ +0.000024] legacy_get_tree at fs/fs_context.c:662 [ +0.000002] vfs_get_tree at fs/super.c:1771 [ +0.000002] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:181 [ +0.000002] do_new_mount at fs/namespace.c:3349 (inlined by) path_mount at fs/namespace.c:3675 [ +0.000002] do_mount at fs/namespace.c:3689 (inlined by) __do_sys_mount at fs/namespace.c:3897 (inlined by) __se_sys_mount at fs/namespace.c:3874 (inlined by) __x64_sys_mount at fs/namespace.c:3874 [ +0.000002] do_syscall_x64 at arch/x86/entry/common.c:52 (inlined by) do_syscall_64 at arch/x86/entry/common.c:83 [ +0.000003] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:181 [ +0.000001] ? fdput at include/linux/file.h:46 (inlined by) fdput_pos at include/linux/file.h:81 (inlined by) ksys_write at fs/read_write.c:640 (inlined by) __do_sys_write at fs/read_write.c:649 (inlined by) __se_sys_write at fs/read_write.c:646 (inlined by) __x64_sys_write at fs/read_write.c:646 [ +0.000002] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:181 [ +0.000002] ? syscall_exit_to_user_mode at kernel/entry/common.c:299 [ +0.000001] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:181 [ +0.000002] ? _static_cpu_has at arch/x86/include/asm/cpufeature.h:171 (inlined by) do_syscall_64 at arch/x86/entry/common.c:98 [ +0.000001] ? _static_cpu_has at arch/x86/include/asm/cpufeature.h:171 (inlined by) do_syscall_64 at arch/x86/entry/common.c:98 [ +0.000002] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:181 [ +0.000001] ? arch_local_irq_disable at arch/x86/include/asm/paravirt.h:695 (inlined by) handle_page_fault at arch/x86/mm/fault.c:1515 (inlined by) exc_page_fault at arch/x86/mm/fault.c:1563 [ +0.000002] entry_SYSCALL_64_after_hwframe at arch/x86/entry/entry_64.S:129 [ +0.000002] RIP: 0033:0x7f1f9f267d2e [ +0.000012] RSP: 002b:00007ffda0b30cb8 EFLAGS: 00000206 ORIG_RAX: 00000000000000a5 [ +0.000002] RAX: ffffffffffffffda RBX: 00005eee92470d80 RCX: 00007f1f9f267d2e [ +0.000001] RDX: 00005eee9246e590 RSI: 00005eee9246ea00 RDI: 00005eee92470e80 [ +0.000001] RBP: 00005eee92470d80 R08: 00005eee92470d80 R09: 0000000000000007 [ +0.000000] R10: 0000000002200000 R11: 0000000000000206 R12: 00005eee92470e80 [ +0.000001] R13: 0000000000000090 R14: 00005eee924716b0 R15: 0000000000000004 [ +0.000003] ```
Snogard commented 5 months ago

After a night of sleep i recompiled the kernel without changing anything and... faddr2line worked on the bcachefs module.

here is the complete version of the addressed dmesg, i hope i did it right.

addressed dmesg ``` [Feb 5 15:31] INFO: task kworker/u16:1:13 blocked for more than 122 seconds. [ +0.000005] Not tainted 6.7.3-zen1-2-zen #1 [ +0.000002] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ +0.000001] task:kworker/u16:1 state:D stack:0 pid:13 tgid:13 ppid:2 flags:0x00004000 [ +0.000004] Workqueue: btree_update btree_interior_update_work [bcachefs] [ +0.000043] Call Trace: [ +0.000001] [ +0.000003] context_switch at kernel/sched/core.c:5376 (inlined by) __schedule at kernel/sched/core.c:6688 [ +0.000006] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:181 [ +0.000003] ? __bch2_time_stats_update at fs/bcachefs/util.c:403 (discriminator 1) [ +0.000040] __preempt_count_sub at arch/x86/include/asm/preempt.h:85 (discriminator 13) (inlined by) __schedule_loop at kernel/sched/core.c:6764 (discriminator 13) (inlined by) schedule at kernel/sched/core.c:6778 (discriminator 13) [ +0.000003] __closure_sync at lib/closure.c:132 (discriminator 4) [ +0.000003] closure_sync at ./include/linux/closure.h:194 (inlined by) bch2_btree_update_start at fs/bcachefs/btree_update_interior.c:1168 [ +0.000047] ? closure_sync_fn at lib/closure.c:112 [ +0.000004] ? bch2_btree_split_leaf at fs/bcachefs/btree_update_interior.c:1738 [ +0.000030] bch2_btree_split_leaf at fs/bcachefs/btree_update_interior.c:1738 [ +0.000030] ? bch2_trans_commit at fs/bcachefs/btree_update.h:137 (inlined by) btree_update_nodes_written at fs/bcachefs/btree_update_interior.c:646 (inlined by) btree_interior_update_work at fs/bcachefs/btree_update_interior.c:778 [ +0.000027] bch2_trans_commit_error at fs/bcachefs/btree_trans_commit.c:917 [ +0.000022] __bch2_trans_commit at fs/bcachefs/btree_trans_commit.c:1157 (discriminator 1)] [ +0.000022] bch2_trans_commit at fs/bcachefs/btree_update.h:137 (inlined by) btree_update_nodes_written at fs/bcachefs/btree_update_interior.c:646 (inlined by) btree_interior_update_work at fs/bcachefs/btree_update_interior.c:778 [ +0.000021] process_one_work at kernel/workqueue.c:2642 [ +0.000002] process_scheduled_works at kernel/workqueue.c:2709 (discriminator 2) (inlined by) worker_thread at kernel/workqueue.c:2796 (discriminator 2) [ +0.000002] ? worker_thread at kernel/workqueue.c:2742 [ +0.000002] kthread at kernel/kthread.c:388 [ +0.000001] ? kthread at kernel/kthread.c:341 [ +0.000002] ret_from_fork at arch/x86/kernel/process.c:147 [ +0.000002] ? kthread at kernel/kthread.c:341 [ +0.000002] ret_from_fork_asm at arch/x86/entry/entry_64.S:250 [ +0.000004] [ +0.000040] INFO: task bcachefs:1360 blocked for more than 122 seconds. [ +0.000001] Not tainted 6.7.3-zen1-2-zen #1 [ +0.000001] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ +0.000000] task:bcachefs state:D stack:0 pid:1360 tgid:1360 ppid:1359 flags:0x00004002 [ +0.000002] Call Trace: [ +0.000001] [ +0.000001] context_switch at kernel/sched/core.c:5376 (inlined by) __schedule at kernel/sched/core.c:6688 [ +0.000002] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:181 [ +0.000002] ? __bch2_time_stats_update at fs/bcachefs/util.c:403 (discriminator 1) [ +0.000023] __preempt_count_sub at arch/x86/include/asm/preempt.h:85 (discriminator 13) (inlined by) __schedule_loop at kernel/sched/core.c:6764 (discriminator 13) (inlined by) schedule at kernel/sched/core.c:6778 (discriminator 13) [ +0.000002] __closure_sync at lib/closure.c:132 (discriminator 4) [ +0.000001] closure_sync at ./include/linux/closure.h:194 (inlined by) bch2_btree_update_start at fs/bcachefs/btree_update_interior.c:116 [ +0.000021] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:181 [ +0.000001] ? bkey_iter_cmp_p_or_unp at fs/bcachefs/bset.h:400 (inlined by) bch2_bset_search_linear at fs/bcachefs/bset.c:1206 (inlined by) bch2_btree_node_iter_init at fs/bcachefs/bset.c:1351 [ +0.000022] ? closure_sync_fn at lib/closure.c:112 [ +0.000002] ? bch2_btree_split_leaf at fs/bcachefs/btree_update_interior.c:1738 [ +0.000020] bch2_btree_split_leaf at fs/bcachefs/btree_update_interior.c:1738 [ +0.000019] ? bch2_trans_commit at fs/bcachefs/btree_update.h:137 (inlined by) bch2_journal_replay at fs/bcachefs/recovery.c:180 [ +0.000026] bch2_trans_commit_error at fs/bcachefs/btree_trans_commit.c:917 [ +0.000021] __bch2_trans_commit at fs/bcachefs/btree_trans_commit.c:1157 (discriminator 1)] [ +0.000022] bch2_trans_commit at fs/bcachefs/btree_update.h:137 (inlined by) bch2_journal_replay at fs/bcachefs/recovery.c:180 [ +0.000023] bch2_run_recovery_pass at fs/bcachefs/recovery.c:655 (inlined by) bch2_run_recovery_passes at fs/bcachefs/recovery.c:672 (inlined by) bch2_fs_recovery at fs/bcachefs/recovery.c:918 [ +0.000021] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:181 [ +0.000001] ? bch2_printbuf_exit at fs/bcachefs/printbuf.c:111 (discriminator 1) [ +0.000022] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:181 [ +0.000001] ? print_mount_opts at fs/bcachefs/super.c:945 [ +0.000024] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:181 [ +0.000002] bch2_fs_start at fs/bcachefs/super.c:978 [ +0.000021] bch2_fs_open at fs/bcachefs/super.c:1968 [ +0.000025] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:181 [ +0.000003] ? bch2_mount at fs/bcachefs/fs.c:1863 [ +0.000025] bch2_mount at fs/bcachefs/fs.c:1863 [ +0.000024] legacy_get_tree at fs/fs_context.c:662 [ +0.000002] vfs_get_tree at fs/super.c:1771 [ +0.000002] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:181 [ +0.000002] do_new_mount at fs/namespace.c:3349 (inlined by) path_mount at fs/namespace.c:3675 [ +0.000002] do_mount at fs/namespace.c:3689 (inlined by) __do_sys_mount at fs/namespace.c:3897 (inlined by) __se_sys_mount at fs/namespace.c:3874 (inlined by) __x64_sys_mount at fs/namespace.c:3874 [ +0.000002] do_syscall_x64 at arch/x86/entry/common.c:52 (inlined by) do_syscall_64 at arch/x86/entry/common.c:83 [ +0.000003] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:181 [ +0.000001] ? fdput at include/linux/file.h:46 (inlined by) fdput_pos at include/linux/file.h:81 (inlined by) ksys_write at fs/read_write.c:640 (inlined by) __do_sys_write at fs/read_write.c:649 (inlined by) __se_sys_write at fs/read_write.c:646 (inlined by) __x64_sys_write at fs/read_write.c:646 [ +0.000002] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:181 [ +0.000002] ? syscall_exit_to_user_mode at kernel/entry/common.c:299 [ +0.000001] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:181 [ +0.000002] ? _static_cpu_has at arch/x86/include/asm/cpufeature.h:171 (inlined by) do_syscall_64 at arch/x86/entry/common.c:98 [ +0.000001] ? _static_cpu_has at arch/x86/include/asm/cpufeature.h:171 (inlined by) do_syscall_64 at arch/x86/entry/common.c:98 [ +0.000002] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:181 [ +0.000001] ? arch_local_irq_disable at arch/x86/include/asm/paravirt.h:695 (inlined by) handle_page_fault at arch/x86/mm/fault.c:1515 (inlined by) exc_page_fault at arch/x86/mm/fault.c:1563 [ +0.000002] entry_SYSCALL_64_after_hwframe at arch/x86/entry/entry_64.S:129 [ +0.000002] RIP: 0033:0x7f1f9f267d2e [ +0.000012] RSP: 002b:00007ffda0b30cb8 EFLAGS: 00000206 ORIG_RAX: 00000000000000a5 [ +0.000002] RAX: ffffffffffffffda RBX: 00005eee92470d80 RCX: 00007f1f9f267d2e [ +0.000001] RDX: 00005eee9246e590 RSI: 00005eee9246ea00 RDI: 00005eee92470e80 [ +0.000001] RBP: 00005eee92470d80 R08: 00005eee92470d80 R09: 0000000000000007 [ +0.000000] R10: 0000000002200000 R11: 0000000000000206 R12: 00005eee92470e80 [ +0.000001] R13: 0000000000000090 R14: 00005eee924716b0 R15: 0000000000000004 [ +0.000003] ```
Snogard commented 5 months ago

any news?

lorenzpmeier commented 5 months ago

Hi @Snogard and @koverstreet , I am facing identical issues.

Setup:

What happened?

How I tried to recover the data:

Snogard commented 5 months ago

@lorenzpmeier you may have to faddr2line the dmesg output like i did.

lorenzpmeier commented 5 months ago

Hi @Snogard, I get the same error you got, ERROR: CONFIG_DEBUG_INFO not enabled when running 

/usr/lib/modules/6.7.5-zen1-1-zen/build/scripts/faddr2line /usr/lib/modules/6.7.5-zen1-1-zen/build/vmlinux /usr/lib/modules/6.7.5-zen1-1-zen/kernel/fs/bcachefs/bcachefs.ko.zst

from root. Swapping kernel to 6.7.4-artix1-1 doesn't change the error.

What am I missing?

Snogard commented 5 months ago

You must recompile your kernel disabling the stripping from the pkgbuild (from line 187 to 203) and set INSTALL_MOD_STRIP=0.

Then for every line run something like this: /path/to/faddr2line /path/to/module.ko __schedule+0xcaa/0x1950

lorenzpmeier commented 5 months ago

Thanks. I checked via cat /proc/config.gz | gunzip | grep CONFIG_DEBUG_INFO and got CONFIG_DEBUG_INFO=y which looked sufficient.

I will

I'll report back tomorrow.

koverstreet commented 5 months ago

Sorry for taking so long to look at this; looking at the backtraces, we're probably blocked on the allocator.

Can you check /sys/fs/bcachefs/*/dev-0/alloc_debug?

lorenzpmeier commented 5 months ago

/sys/fs/bcachefs/*/dev-0/alloc_debug output:

free                   0               0               0
sb                     0               0               0
journal                0               0               0
btree                  0               0               0
user                   0               0               0
cached                 0               0               0
parity                 0               0               0
stripe                 0               0               0
need_gc_gens           0               0               0
need_discard           0               0               0
ec                     0

reserves:
stripe           1192156
normal            596092
copygc                28
btree                 14
btree_copygc           0
reclaim                0

freelist_wait           empty
open buckets allocated  1
open buckets this dev   0
open buckets total      1024
open_buckets_wait       empty
open_buckets_btree      0
open_buckets_user       0
buckets_to_invalidate   0
btree reserve cache     0

Linux build failed, OOM, I'll swap server and rebuild on a bigger machine. I'll report back ASAP.

@koverstreet Thanks in advance!

Update 2024-02-23 11:17 UTC+0: On building Linux as mentioned above, following issue arises @Snogard. I ran BUILDDIR=/home/lorenz/linux-zen/BUILDDIR makepkg --skippgpcheck

==> Entering fakeroot environment...
==> Starting package_linux-zen()...
Installing boot image...
Installing modules...
  SYMLINK /home/lorenz/linux-zen/BUILDDIR/linux-zen/pkg/linux-zen/usr/lib/modules/6.7.5-zen1-1-zen/build
  INSTALL /home/lorenz/linux-zen/BUILDDIR/linux-zen/pkg/linux-zen/usr/lib/modules/6.7.5-zen1-1-zen/modules.order
  INSTALL /home/lorenz/linux-zen/BUILDDIR/linux-zen/pkg/linux-zen/usr/lib/modules/6.7.5-zen1-1-zen/modules.builtin
  INSTALL /home/lorenz/linux-zen/BUILDDIR/linux-zen/pkg/linux-zen/usr/lib/modules/6.7.5-zen1-1-zen/modules.builtin.modinfo
  INSTALL /home/lorenz/linux-zen/BUILDDIR/linux-zen/pkg/linux-zen/usr/lib/modules/6.7.5-zen1-1-zen/kernel/arch/x86/events/amd/power.ko
  STRIP   /home/lorenz/linux-zen/BUILDDIR/linux-zen/pkg/linux-zen/usr/lib/modules/6.7.5-zen1-1-zen/kernel/arch/x86/events/amd/power.ko
strip: '0': No such file
make[2]: *** [scripts/Makefile.modinst:120: /home/lorenz/linux-zen/BUILDDIR/linux-zen/pkg/linux-zen/usr/lib/modules/6.7.5-zen1-1-zen/kernel/arch/x86/events/amd/power.ko] Error 1
make[2]: *** Deleting file '/home/lorenz/linux-zen/BUILDDIR/linux-zen/pkg/linux-zen/usr/lib/modules/6.7.5-zen1-1-zen/kernel/arch/x86/events/amd/power.ko'
make[1]: *** [/home/lorenz/linux-zen/BUILDDIR/linux-zen/src/linux-6.7.5/Makefile:1817: modules_install] Error 2
make: *** [Makefile:234: __sub-make] Error 2
==> ERROR: A failure occurred in package_linux-zen().
    Aborting...
lorenzpmeier commented 5 months ago

While I am banging my head against compiling linux to get richer pointers, I stumbled across some interesting new errors.

After unlocking my drives, I tried to run fsck to reconstruct the alloc tree, i.e. bcachefs fsck -vnR /dev/sda:/dev/sdb:/dev/nvme0n1 which leads to:

bcachefs fsck -nrv /dev/sda:/dev/sdb:/dev/nvme0n1
bch2_dev_in_fs() Split brain detected between /dev/nvme0n1 and /dev/sda:
/dev/sda believes seq of /dev/nvme0n1 to be 65, but /dev/nvme0n1 has 94
Not using /dev/nvme0n1
bch2_dev_in_fs() Split brain detected between /dev/sdb and /dev/sda:
/dev/sda believes seq of /dev/sdb to be 65, but /dev/sdb has 97
Not using /dev/sdb
bch2_fs_open() bch_fs_open err opening /dev/sda: insufficient_devices_to_start
bch2_dev_in_fs() Split brain detected between /dev/nvme0n1 and /dev/sda:
/dev/sda believes seq of /dev/nvme0n1 to be 65, but /dev/nvme0n1 has 94
Not using /dev/nvme0n1
bch2_dev_in_fs() Split brain detected between /dev/sdb and /dev/sda:
/dev/sda believes seq of /dev/sdb to be 65, but /dev/sdb has 97
Not using /dev/sdb
insufficient devices online (0) for replicas entry user: 1/2 [1 2]
bch2_fs_open() bch_fs_open err opening /dev/sda: insufficient_devices_to_start
shutting down
shutdown complete

So I thought, hey, might as well use the -k switch to Use the in-kernel fsck implementation, which leads to bcachefs fsck -kvnR /dev/sda:/dev/sdb:/dev/nvme0n1 and the output

BCH_IOCTL_FSCK_OFFLINE error: Operation not permitted

dmesg stays empty.

@koverstreet how FUBAR is my situation?

koverstreet commented 4 months ago

lorenzpmeier - if you've still got this fs there's an option in my master branch, no_splitbrain_check, which will let you mount

Sorry it took so long to get to this!

lorenzpmeier commented 4 months ago

Hi Kent,

I ran bcachefs mount /dev/sdc:/dev/sdb:/dev/nvme0n1 /mnt -o ro,fsck,no_splitbrain_check but it did not work. Following error

INFO - bcachefs::key: Attempting to unlock master key for filesystem b1df1cb0-af7f-4ab0-8b11-18d22e514108, using unlock policy Ask
Enter passphrase:
INFO - bcachefs::commands::cmd_mount: mounting with params: device: /dev/sdc:/dev/sdb:/dev/nvme0n1, target: /mnt,  verbose,ro,fsck,no_splitbrain_check
DEBUG - bcachefs::commands::cmd_mount: parsing mount options: verbose,ro,fsck,no_splitbrain_check
INFO - bcachefs::commands::cmd_mount: mounting bcachefs filesystem, /mnt
INFO - bcachefs::commands::cmd_mount: mounting filesystem
ERROR - bcachefs::commands::cmd_mount: Fatal error: Invalid argument

and got the following dmesg:

[Mar12 00:23] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): mounting version 1.6: (unknown version) opts=ro,metadata_replicas=2,data_replicas=2,compression=zstd,metadata_target=/dev/sdb,foreground_target=ssd,background_target=bg_group,promote_target=ssd,fsck
[  +0.000008] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): recovering from unclean shutdown
[  +0.000005] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): superblock requires following recovery passes to be run:
                check_subvols,check_dirents
[  +0.000003] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): Version downgrade required:

[  +0.000006] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): Version upgrade from 1.3: rebalance_work to 1.6: (unknown version) incomplete
              Doing compatible version upgrade from 1.3: rebalance_work to 1.6: (unknown version)

[Mar12 00:24] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): journal read done, replaying entries 3449362-3453196
[  +0.000007] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): dropped unflushed entries 3453197-3453197
[  +1.400036] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): alloc_read... done
[  +0.473012] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): stripes_read... done
[  +0.000010] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): snapshots_read... done
[  +0.000088] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): check_allocations...
[Mar12 00:52] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): error validating btree node on sdc at btree backpointers level 0/2
[  +0.000003]   u64s 12 type btree_ptr_v2 2:3835760607232:0 len 0 ver 0: seq f63e0ab2ba2c7878 written 72 min_key 2:3673709256704:1 durability: 2 ptr: 0:199660:512 gen 4 ptr: 1:10708637:512 gen 9
[  +0.000002]   node offset 0: got wrong btree node (seq ceda499762806338 want f63e0ab2ba2c7878)
[  +0.000098] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): retrying read
[  +0.009070] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): error validating btree node on sdb at btree backpointers level 0/2
[  +0.000004]   u64s 12 type btree_ptr_v2 2:3835760607232:0 len 0 ver 0: seq f63e0ab2ba2c7878 written 72 min_key 2:3673709256704:1 durability: 2 ptr: 0:199660:512 gen 4 ptr: 1:10708637:512 gen 9
[  +0.000001]   node offset 0: got wrong btree node (seq ceda499762806338 want f63e0ab2ba2c7878)
[  +0.003844] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): running explicit recovery pass check_topology (4), currently at check_allocations (5)
[  +0.000008] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): retry success
[  +0.000039] Unreadable btree node at btree backpointers level 0:
[  +0.000002]   u64s 12 type btree_ptr_v2 2:3835760607232:0 len 0 ver 0: seq f63e0ab2ba2c7878 written 72 min_key 2:3673709256704:1 durability: 2 ptr: 0:199660:512 gen 4 ptr: 1:10708637:512 gen 9, exiting
[  +0.003412] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): Unable to continue, halting
[  +0.001094] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): bch2_gc_btree_init(): error fsck_errors_not_fixed
[  +0.001097] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): bch2_gc_btrees(): error fsck_errors_not_fixed
[  +0.047758] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): bch2_gc(): error fsck_errors_not_fixed
[  +0.001401] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): bch2_fs_recovery(): error fsck_errors_not_fixed
[  +0.000589] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): bch2_fs_start(): error starting filesystem fsck_errors_not_fixed

The Fatal error: Invalid argument might hint towards something missing in my version, bcachefs version shows 1.6.4.

koverstreet commented 4 months ago

You need to also include the fix_errors option

lorenzpmeier commented 4 months ago

Thanks!

Thus I ran bcachefs mount /dev/sdc:/dev/sdb:/dev/nvme0n1 /mnt -o ro,fsck,no_splitbrain_check,fix_errors, leading to

INFO - bcachefs::commands::cmd_mount: Successfully mounted

and a dmesg reporting

[Mar13 11:20] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): mounting version 1.6: (unknown version) opts=ro,metadata_replicas=2,data_replicas=2,compression=zstd,metadata_target=/dev/sdb,foreground_target=ssd,background_target=bg_group,promote_target=ssd,fsck,fix_errors=yes
[  +0.000005] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): recovering from unclean shutdown
[  +0.000003] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): superblock requires following recovery passes to be run:
                check_subvols,check_dirents
[  +0.000001] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): Version downgrade required:

[  +0.000004] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): Version upgrade from 1.3: rebalance_work to 1.6: (unknown version) incomplete
              Doing compatible version upgrade from 1.3: rebalance_work to 1.6: (unknown version)

[Mar13 11:21] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): journal read done, replaying entries 3449362-3453196
[  +0.000004] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): dropped unflushed entries 3453197-3453197
[  +1.162372] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): bch2_journal_reclaim_start(): error creating journal reclaim thread EINTR
[  +0.000325] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): bch2_fs_recovery(): error EINTR
[  +0.000023] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): bch2_fs_start(): error starting filesystem EINTR
[Mar13 11:22] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): mounting version 1.6: (unknown version) opts=ro,metadata_replicas=2,data_replicas=2,compression=zstd,metadata_target=/dev/sdb,foreground_target=ssd,background_target=bg_group,promote_target=ssd,fsck,fix_errors=yes
[  +0.000005] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): recovering from unclean shutdown
[  +0.000002] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): superblock requires following recovery passes to be run:
                check_subvols,check_dirents
[  +0.000002] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): Version downgrade required:

[  +0.000003] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): Version upgrade from 1.3: rebalance_work to 1.6: (unknown version) incomplete
              Doing compatible version upgrade from 1.3: rebalance_work to 1.6: (unknown version)

[Mar13 11:23] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): journal read done, replaying entries 3449362-3453196
[  +0.000007] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): dropped unflushed entries 3453197-3453197
[  +1.416716] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): alloc_read... done
[  +0.543449] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): stripes_read... done
[  +0.000010] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): snapshots_read... done
[  +0.000086] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): check_allocations...
[Mar13 11:53] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): error validating btree node on sdb at btree backpointers level 0/2
[  +0.000003]   u64s 12 type btree_ptr_v2 2:3835760607232:0 len 0 ver 0: seq f63e0ab2ba2c7878 written 72 min_key 2:3673709256704:1 durability: 2 ptr: 0:199660:512 gen 4 ptr: 1:10708637:512 gen 9
[  +0.000002]   node offset 0: got wrong btree node (seq ceda499762806338 want f63e0ab2ba2c7878)
[  +0.000097] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): retrying read
[  +0.009994] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): error validating btree node on sdc at btree backpointers level 0/2
[  +0.000003]   u64s 12 type btree_ptr_v2 2:3835760607232:0 len 0 ver 0: seq f63e0ab2ba2c7878 written 72 min_key 2:3673709256704:1 durability: 2 ptr: 0:199660:512 gen 4 ptr: 1:10708637:512 gen 9
[  +0.000002]   node offset 0: got wrong btree node (seq ceda499762806338 want f63e0ab2ba2c7878)
[  +0.000090] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): running explicit recovery pass check_topology (4), currently at check_allocations (5)
[  +0.000004] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): retry success
[  +0.000019] Unreadable btree node at btree backpointers level 0:
[  +0.000002]   u64s 12 type btree_ptr_v2 2:3835760607232:0 len 0 ver 0: seq f63e0ab2ba2c7878 written 72 min_key 2:3673709256704:1 durability: 2 ptr: 0:199660:512 gen 4 ptr: 1:10708637:512 gen 9, fixing
[  +0.000052] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): Halting mark and sweep to start topology repair pass
[Mar13 11:54]  done
[  +0.000003] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): check_allocations...
[Mar13 12:22] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): error validating btree node on sdc at btree backpointers level 0/2
[  +0.000003]   u64s 12 type btree_ptr_v2 2:3835760607232:0 len 0 ver 0: seq f63e0ab2ba2c7878 written 72 min_key 2:3673709256704:1 durability: 2 ptr: 0:199660:512 gen 4 ptr: 1:10708637:512 gen 9
[  +0.000002]   node offset 0: got wrong btree node (seq ceda499762806338 want f63e0ab2ba2c7878)
[  +0.000097] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): retrying read
[  +0.009523] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): error validating btree node on sdb at btree backpointers level 0/2
[  +0.000004]   u64s 12 type btree_ptr_v2 2:3835760607232:0 len 0 ver 0: seq f63e0ab2ba2c7878 written 72 min_key 2:3673709256704:1 durability: 2 ptr: 0:199660:512 gen 4 ptr: 1:10708637:512 gen 9
[  +0.000001]   node offset 0: got wrong btree node (seq ceda499762806338 want f63e0ab2ba2c7878)
[  +0.000096] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): retry success
[  +0.000885] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): Halting mark and sweep to start topology repair pass
[Mar13 12:23]  done
[  +0.145624] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): journal_replay...
[  +0.211785] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): going read-write
[  +0.015047] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): bch2_gc_thread_start(): error EINTR
[  +0.000431] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): error starting gc thread
[  +0.000036] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): bch2_journal_replay(): error EINTR
[  +0.000017] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): bch2_fs_recovery(): error EINTR
[  +0.000011] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): bch2_fs_start(): error starting filesystem EINTR
[Mar13 14:59] bcachefs (b1df1cb0-af7f-4ab0-8b11-18d22e514108): error requesting encryption key: ENOKEY

The mount succeeded at 1755hrs, taking ~6h30m, the fsck,fix_errors,no_splitbrain_check option worked, I am transferring files to a new drive as we speak.

If I get more dmesg output, I'll share it, but this seems to have solved at least the mounting and access issue.

Thanks @koverstreet ! Great work and thanks a bunch for your support.

Biggest learning for me: Don't put your backups only on old hard drives that die on replay.

Snogard commented 4 months ago

@koverstreet do you think i should try too? or is my problem different?

koverstreet commented 4 months ago

@Snogard yours looks different.

When it hangs, grab /sys/fs/bcachefs//dev-0/alloc_debug and post that.

I've got another filesystem to debug that's showing something similar, I'll work on that today as well

koverstreet commented 3 months ago

Ok, I've got a fix for deadlocks during journal replay in my master branch. Can whoever is still hitting that try and report back?

I think all bugs mentioned in this thread are fixed now; please reopen this or a new bug if needed.

Snogard commented 3 months ago

Sorry @koverstreet for the late reply but i've been without internet for the past weeks. Anyway i compiled the kernel from your last commit and my filesystem still hangs.

Here is the alloc for all three devices:

dev-0 ``` cat /sys/fs/bcachefs/d6020e9b-770a-4aa5-a4af-389f4003b650/dev-0/alloc_debug buckets sectors fragmented free 1895087 0 0 sb 7 6152 1016 journal 8192 8388608 0 btree 186655 111570432 79564288 user 13145427 13321947122 138970126 cached 26033 13916696 0 parity 0 0 0 stripe 0 0 0 need_gc_gens 0 0 0 need_discard 369 0 0 reserves: stripe 476958 normal 238493 copygc 28 btree 14 btree_copygc 0 reclaim 0 interior_updates 0 freelist_wait empty open buckets allocated 1024 open buckets this dev 679 open buckets total 1024 open_buckets_wait waiting open_buckets_btree 1023 open_buckets_user 0 buckets_to_invalidate 0 btree reserve cache 0 ```
dev-1 ``` cat /sys/fs/bcachefs/d6020e9b-770a-4aa5-a4af-389f4003b650/dev-1/alloc_debug buckets sectors fragmented free 1897846 0 0 sb 7 6152 1016 journal 8192 8388608 0 btree 183894 108941312 79366144 user 13145428 13321947546 138970726 cached 26027 13927360 0 parity 0 0 0 stripe 0 0 0 need_gc_gens 0 0 0 need_discard 376 0 0 reserves: stripe 476958 normal 238493 copygc 28 btree 14 btree_copygc 0 reclaim 0 interior_updates 0 freelist_wait empty open buckets allocated 1024 open buckets this dev 0 open buckets total 1024 open_buckets_wait waiting open_buckets_btree 1023 open_buckets_user 0 buckets_to_invalidate 0 btree reserve cache 0 ```
dev-2 ``` cat /sys/fs/bcachefs/d6020e9b-770a-4aa5-a4af-389f4003b650/dev-2/alloc_debug buckets sectors fragmented free 7607804 0 0 sb 4 6152 2040 journal 8192 16777216 0 btree 1759 3070464 531968 user 13126 26880424 1624 cached 0 0 0 parity 0 0 0 stripe 0 0 0 need_gc_gens 0 0 0 need_discard 0 0 0 reserves: stripe 238478 normal 119246 copygc 14 btree 7 btree_copygc 0 reclaim 0 interior_updates 0 freelist_wait empty open buckets allocated 1024 open buckets this dev 344 open buckets total 1024 open_buckets_wait waiting open_buckets_btree 1023 open_buckets_user 0 buckets_to_invalidate 0 btree reserve cache 0 ```
Snogard commented 3 months ago

github doesn't give me the option to reopen the issue, can you do it instead @koverstreet?

koverstreet commented 3 months ago

reedriley just confirmed - and I got a good look at what's going on, fix should be up in a day or so

Snogard commented 2 months ago

Any progress on this?

Valmar33 commented 2 months ago

Any progress on this?

Are you still having the issue with bcachefs and bcachefs-tools master? I was having this issue on my end until some update seemingly fixed the problem for now.

Snogard commented 2 months ago

last time i checked was april 2, i was waiting for an update before testing again. anyway i'll try again as soon as possible and report back

Snogard commented 2 months ago

Yep, still having problems... I don't have time now to use faddr2line right now, but in the meantime here is the dmesg. I'll try to post the results of faddr2line tomorrow at least.

dmesg ``` [ +0.341804] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): mounting version 1.6: btree_subvolume_children opts=metadata_replicas=2,data_replicas=2,compression=zstd,metadata_target=hdd,background_target=hdd,verbose,fsck,fix_errors=yes [ +0.000006] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): recovering from unclean shutdown [ +0.000002] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): superblock requires following recovery passes to be run: check_subvols,check_dirents [ +0.000003] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): Version upgrade from 1.3: rebalance_work to 1.6: btree_subvolume_children incomplete Doing compatible version upgrade from 1.3: rebalance_work to 1.7: mi_btree_bitmap running recovery passes: check_allocations [ +0.009089] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): starting journal read [ +17.443927] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): journal read done on device sdb, ret 0 [ +17.206896] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): journal read done on device sdc, ret 0 [ +0.000029] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): journal read done, replaying entries 45201688-45202952 [ +0.000113] journal entries 45202101-45202108 missing! (replaying 45201688-45202952) [ +0.000002] prev at 0:1041:432 (sector 1071536) 2:606:992 (sector 1248224) size 8 [ +0.000001] next at 0:1041:440 (sector 1071544) 2:606:1000 (sector 1248232), continuing [ +0.000008] journal entries 45202143-45202150 missing! (replaying 45201688-45202952) [ +0.000001] prev at 0:1043:240 (sector 1073392) 2:607:816 (sector 1250096) size 16 [ +0.000001] next at 0:1043:256 (sector 1073408) 2:607:832 (sector 1250112), continuing [ +0.000007] journal entries 45202196-45202203 missing! (replaying 45201688-45202952) [ +0.000001] prev at 0:1045:344 (sector 1075544) 2:608:904 (sector 1252232) size 16 [ +0.000000] next at 0:1045:360 (sector 1075560) 2:608:920 (sector 1252248), continuing [ +0.000007] journal entries 45202254-45202261 missing! (replaying 45201688-45202952) [ +0.000001] prev at 0:1047:496 (sector 1077744) 2:609:1096 (sector 1254472) size 8 [ +0.000001] next at 0:1047:504 (sector 1077752) 2:609:1104 (sector 1254480), continuing [ +0.000006] journal entries 45202300-45202307 missing! (replaying 45201688-45202952) [ +0.000001] prev at 0:1049:344 (sector 1079640) 2:610:960 (sector 1256384) size 8 [ +0.000001] next at 0:1049:352 (sector 1079648) 2:610:968 (sector 1256392), continuing [ +0.000007] journal entries 45202358-45202365 missing! (replaying 45201688-45202952) [ +0.000001] prev at 0:1051:992 (sector 1082336) 2:611:1576 (sector 1259048) size 16 [ +0.000000] next at 0:1052:0 (sector 1082368) 2:611:1592 (sector 1259064), continuing [ +0.630583] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): Journal keys: 944320 read, 541751 after sorting and compacting [ +0.396343] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): alloc_read... done [ +1.199425] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): stripes_read... done [ +0.000006] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): snapshots_read... done [ +0.000005] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): check_allocations... [May18 10:27] zsh (842): drop_caches: 3 [ +35.002875] done [ +0.547018] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): going read-write [ +0.001825] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): journal_replay... [May18 10:34] INFO: task kworker/4:1:175 blocked for more than 122 seconds. [ +0.000005] Not tainted 6.9.0-rc6-1-bcachefs-git-00178-g07f9a27f1969 #1 [ +0.000001] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ +0.000002] task:kworker/4:1 state:D stack:0 pid:175 tgid:175 ppid:2 flags:0x00004000 [ +0.000005] Workqueue: bcachefs_write_ref bch2_do_discards_work [bcachefs] [ +0.000027] Call Trace: [ +0.000001] [ +0.000004] __schedule+0x3c7/0x1510 [ +0.000009] schedule+0x27/0xf0 [ +0.000002] __closure_sync+0x7e/0x140 [ +0.000004] bch2_btree_update_start+0x83d/0x870 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000036] ? __pfx_closure_sync_fn+0x10/0x10 [ +0.000005] ? bch2_discard_one_bucket+0x865/0x950 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000022] bch2_btree_split_leaf+0x57/0x1b0 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000029] ? bch2_discard_one_bucket+0x865/0x950 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000021] bch2_trans_commit_error+0x1fb/0x4e0 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000027] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000003] ? six_trylock_ip+0x1f/0x50 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000036] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000003] __bch2_trans_commit+0x1097/0x1780 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000028] ? __entry_text_end+0x101e45/0x101e49 [ +0.000005] bch2_discard_one_bucket+0x865/0x950 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000026] ? bch2_do_discards_work+0x20d/0x3c0 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000021] bch2_do_discards_work+0x20d/0x3c0 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000026] process_one_work+0x18b/0x350 [ +0.000004] worker_thread+0x2eb/0x410 [ +0.000004] ? __pfx_worker_thread+0x10/0x10 [ +0.000002] kthread+0xcf/0x100 [ +0.000003] ? __pfx_kthread+0x10/0x10 [ +0.000003] ret_from_fork+0x31/0x50 [ +0.000003] ? __pfx_kthread+0x10/0x10 [ +0.000002] ret_from_fork_asm+0x1a/0x30 [ +0.000005] [ +0.000009] INFO: task bcachefs:816 blocked for more than 122 seconds. [ +0.000002] Not tainted 6.9.0-rc6-1-bcachefs-git-00178-g07f9a27f1969 #1 [ +0.000001] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ +0.000001] task:bcachefs state:D stack:0 pid:816 tgid:816 ppid:815 flags:0x00004002 [ +0.000003] Call Trace: [ +0.000001] [ +0.000002] __schedule+0x3c7/0x1510 [ +0.000003] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000004] schedule+0x27/0xf0 [ +0.000003] __closure_sync+0x7e/0x140 [ +0.000003] bch2_btree_update_start+0x83d/0x870 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000028] ? __pfx_closure_sync_fn+0x10/0x10 [ +0.000003] ? bch2_journal_replay+0x168/0x6b0 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000033] bch2_btree_split_leaf+0x57/0x1b0 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000028] ? bch2_journal_replay+0x168/0x6b0 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000028] bch2_trans_commit_error+0x1fb/0x4e0 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000027] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000002] ? six_trylock_ip+0x1f/0x50 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000029] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000003] __bch2_trans_commit+0x1097/0x1780 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000026] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000005] bch2_journal_replay+0x168/0x6b0 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000027] ? __bch2_print+0x85/0xf0 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000032] bch2_run_recovery_pass+0x35/0xa0 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000038] bch2_run_recovery_passes+0xae/0x1a0 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000026] bch2_fs_recovery+0xdb3/0x1310 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000028] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000002] ? __bch2_sb_field_resize+0xbb/0x100 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000030] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000001] ? bch2_recalc_capacity+0xfe/0x340 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000024] bch2_fs_start+0x1a4/0x270 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000028] bch2_fs_open+0xf41/0x15c0 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000037] ? bch2_mount+0x591/0x770 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000033] bch2_mount+0x591/0x770 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000037] legacy_get_tree+0x28/0x50 [ +0.000003] vfs_get_tree+0x26/0xd0 [ +0.000003] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000002] path_mount+0x489/0xae0 [ +0.000004] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000002] __x64_sys_mount+0x11c/0x150 [ +0.000004] do_syscall_64+0x82/0x160 [ +0.000004] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000002] ? remove_wait_queue+0x1a/0x70 [ +0.000002] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000001] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000002] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000001] ? n_tty_write+0x372/0x520 [ +0.000004] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000001] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000002] ? __wake_up+0x44/0x60 [ +0.000002] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000002] ? file_tty_write.isra.0+0x20f/0x2c0 [ +0.000003] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000002] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000002] ? vfs_write+0x293/0x460 [ +0.000006] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000003] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000001] ? syscall_exit_to_user_mode+0x75/0x210 [ +0.000002] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000001] ? do_syscall_64+0x8e/0x160 [ +0.000003] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000001] ? do_user_addr_fault+0x34e/0x640 [ +0.000002] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000001] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000002] entry_SYSCALL_64_after_hwframe+0x76/0x7e [ +0.000002] RIP: 0033:0x7f771395534e [ +0.000014] RSP: 002b:00007ffc1ae982d8 EFLAGS: 00000202 ORIG_RAX: 00000000000000a5 [ +0.000002] RAX: ffffffffffffffda RBX: 0000562c2b8a1d20 RCX: 00007f771395534e [ +0.000001] RDX: 0000562c2b881a10 RSI: 0000562c2b881b70 RDI: 0000562c2b8811e0 [ +0.000001] RBP: 00007ffc1ae986f0 R08: 0000562c2b8a1d20 R09: 0000000000000007 [ +0.000001] R10: 0000000002200000 R11: 0000000000000202 R12: 0000562c2b881b70 [ +0.000001] R13: 0000000000000009 R14: 000000000000000d R15: 0000562c2b881a10 [ +0.000003] [ +0.000003] INFO: task bch-reclaim/d60:857 blocked for more than 122 seconds. [ +0.000001] Not tainted 6.9.0-rc6-1-bcachefs-git-00178-g07f9a27f1969 #1 [ +0.000001] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ +0.000001] task:bch-reclaim/d60 state:D stack:0 pid:857 tgid:857 ppid:2 flags:0x00004000 [ +0.000002] Call Trace: [ +0.000001] [ +0.000001] __schedule+0x3c7/0x1510 [ +0.000004] schedule+0x27/0xf0 [ +0.000002] __closure_sync+0x7e/0x140 [ +0.000003] bch2_btree_update_start+0x83d/0x870 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000022] ? __pfx_closure_sync_fn+0x10/0x10 [ +0.000003] ? btree_key_cache_flush_pos.constprop.0+0x2bc/0x330 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000021] bch2_btree_split_leaf+0x57/0x1b0 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000021] ? btree_key_cache_flush_pos.constprop.0+0x2bc/0x330 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000020] bch2_trans_commit_error+0x1fb/0x4e0 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000020] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000001] ? six_trylock_ip+0x1f/0x50 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000023] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000002] __bch2_trans_commit+0x1097/0x1780 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000022] btree_key_cache_flush_pos.constprop.0+0x2bc/0x330 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000024] bch2_btree_key_cache_journal_flush+0x144/0x220 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000020] ? bch2_btree_key_cache_journal_flush+0xb0/0x220 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000020] ? __pfx_bch2_btree_key_cache_journal_flush+0x10/0x10 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000020] journal_flush_pins.constprop.0+0x18f/0x2c0 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000026] __bch2_journal_reclaim+0x1d5/0x380 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000023] bch2_journal_reclaim_thread+0x6e/0x160 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000021] ? __pfx_bch2_journal_reclaim_thread+0x10/0x10 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000018] kthread+0xcf/0x100 [ +0.000002] ? __pfx_kthread+0x10/0x10 [ +0.000002] ret_from_fork+0x31/0x50 [ +0.000002] ? __pfx_kthread+0x10/0x10 [ +0.000002] ret_from_fork_asm+0x1a/0x30 [ +0.000003] [ +0.000005] INFO: task kworker/u34:1:942 blocked for more than 122 seconds. [ +0.000001] Not tainted 6.9.0-rc6-1-bcachefs-git-00178-g07f9a27f1969 #1 [ +0.000000] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ +0.000001] task:kworker/u34:1 state:D stack:0 pid:942 tgid:942 ppid:2 flags:0x00004000 [ +0.000002] Workqueue: btree_update btree_interior_update_work [bcachefs] [ +0.000022] Call Trace: [ +0.000001] [ +0.000001] __schedule+0x3c7/0x1510 [ +0.000004] schedule+0x27/0xf0 [ +0.000002] __closure_sync+0x7e/0x140 [ +0.000002] bch2_btree_update_start+0x83d/0x870 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000021] ? __pfx_closure_sync_fn+0x10/0x10 [ +0.000003] ? btree_interior_update_work+0x8cf/0xb10 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000020] bch2_btree_split_leaf+0x57/0x1b0 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000020] ? btree_interior_update_work+0x8cf/0xb10 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000019] bch2_trans_commit_error+0x1fb/0x4e0 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000020] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000001] ? six_trylock_ip+0x1f/0x50 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000022] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000002] __bch2_trans_commit+0x1097/0x1780 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000022] btree_interior_update_work+0x8cf/0xb10 [bcachefs 4322590102f981a303fcc012727780760e2d2e34] [ +0.000022] ? process_one_work+0x18b/0x350 [ +0.000002] ? srso_alias_return_thunk+0x5/0xfbef5 [ +0.000002] process_one_work+0x18b/0x350 [ +0.000002] worker_thread+0x2eb/0x410 [ +0.000003] ? __pfx_worker_thread+0x10/0x10 [ +0.000002] kthread+0xcf/0x100 [ +0.000001] ? __pfx_kthread+0x10/0x10 [ +0.000002] ret_from_fork+0x31/0x50 [ +0.000002] ? __pfx_kthread+0x10/0x10 [ +0.000002] ret_from_fork_asm+0x1a/0x30 [ +0.000003] ```
Snogard commented 2 months ago

Sorry for the delay, here it is.

addessed dmesg ``` [ +0.341804] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): mounting version 1.6: btree_subvolume_children opts=metadata_replicas=2,data_replicas=2,compression=zstd,metadata_target=hdd,background_target=hdd,verbose,fsck,fix_errors=yes [ +0.000006] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): recovering from unclean shutdown [ +0.000002] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): superblock requires following recovery passes to be run: check_subvols,check_dirents [ +0.000003] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): Version upgrade from 1.3: rebalance_work to 1.6: btree_subvolume_children incomplete Doing compatible version upgrade from 1.3: rebalance_work to 1.7: mi_btree_bitmap running recovery passes: check_allocations [ +0.009089] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): starting journal read [ +17.443927] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): journal read done on device sdb, ret 0 [ +17.206896] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): journal read done on device sdc, ret 0 [ +0.000029] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): journal read done, replaying entries 45201688-45202952 [ +0.000113] journal entries 45202101-45202108 missing! (replaying 45201688-45202952) [ +0.000002] prev at 0:1041:432 (sector 1071536) 2:606:992 (sector 1248224) size 8 [ +0.000001] next at 0:1041:440 (sector 1071544) 2:606:1000 (sector 1248232), continuing [ +0.000008] journal entries 45202143-45202150 missing! (replaying 45201688-45202952) [ +0.000001] prev at 0:1043:240 (sector 1073392) 2:607:816 (sector 1250096) size 16 [ +0.000001] next at 0:1043:256 (sector 1073408) 2:607:832 (sector 1250112), continuing [ +0.000007] journal entries 45202196-45202203 missing! (replaying 45201688-45202952) [ +0.000001] prev at 0:1045:344 (sector 1075544) 2:608:904 (sector 1252232) size 16 [ +0.000000] next at 0:1045:360 (sector 1075560) 2:608:920 (sector 1252248), continuing [ +0.000007] journal entries 45202254-45202261 missing! (replaying 45201688-45202952) [ +0.000001] prev at 0:1047:496 (sector 1077744) 2:609:1096 (sector 1254472) size 8 [ +0.000001] next at 0:1047:504 (sector 1077752) 2:609:1104 (sector 1254480), continuing [ +0.000006] journal entries 45202300-45202307 missing! (replaying 45201688-45202952) [ +0.000001] prev at 0:1049:344 (sector 1079640) 2:610:960 (sector 1256384) size 8 [ +0.000001] next at 0:1049:352 (sector 1079648) 2:610:968 (sector 1256392), continuing [ +0.000007] journal entries 45202358-45202365 missing! (replaying 45201688-45202952) [ +0.000001] prev at 0:1051:992 (sector 1082336) 2:611:1576 (sector 1259048) size 16 [ +0.000000] next at 0:1052:0 (sector 1082368) 2:611:1592 (sector 1259064), continuing [ +0.630583] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): Journal keys: 944320 read, 541751 after sorting and compacting [ +0.396343] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): alloc_read... done [ +1.199425] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): stripes_read... done [ +0.000006] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): snapshots_read... done [ +0.000005] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): check_allocations... [May18 10:27] zsh (842): drop_caches: 3 [ +35.002875] done [ +0.547018] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): going read-write [ +0.001825] bcachefs (d6020e9b-770a-4aa5-a4af-389f4003b650): journal_replay... [May18 10:34] INFO: task kworker/4:1:175 blocked for more than 122 seconds. [ +0.000005] Not tainted 6.9.0-rc6-1-bcachefs-git-00178-g07f9a27f1969 #1 [ +0.000001] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ +0.000002] task:kworker/4:1 state:D stack:0 pid:175 tgid:175 ppid:2 flags:0x00004000 [ +0.000005] Workqueue: bcachefs_write_ref bch2_do_discards_work [bcachefs] [ +0.000027] Call Trace: [ +0.000001] [ +0.000004] context_switch at kernel/sched/core.c:5409 (inlined by) __schedule at kernel/sched/core.c:6746 [ +0.000009] __preempt_count_sub at arch/x86/include/asm/preempt.h:84 (discriminator 13) (inlined by) __schedule_loop at kernel/sched/core.c:6824 (discriminator 13) (inlined by) schedule at kernel/sched/core.c:6838 (discriminator 13) [ +0.000002] __closure_sync at lib/closure.c:132 (discriminator 4) [ +0.000004] closure_sync at include/linux/closure.h:194 (inlined by) bch2_btree_update_start at fs/bcachefs/btree_update_interior.c:1267 [ +0.000036] ? closure_sync_fn at lib/closure.c:112 [ +0.000005] ? bch2_trans_commit at fs/bcachefs/btree_update.h:170 (inlined by) bch2_discard_one_bucket at fs/bcachefs/alloc_background.c:1776 [ +0.000022] bch2_btree_split_leaf at fs/bcachefs/btree_update_interior.c:1851 [ +0.000029] ? bch2_trans_commit at fs/bcachefs/btree_update.h:170 (inlined by) bch2_discard_one_bucket at fs/bcachefs/alloc_background.c:1776 [ +0.000021] bch2_trans_commit_error at fs/bcachefs/btree_trans_commit.c:918 [ +0.000027] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:182 [ +0.000003] ? six_trylock_ip at fs/bcachefs/six.c:293 (discriminator 1) [ +0.000036] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:182 [ +0.000003] __bch2_trans_commit at fs/bcachefs/btree_trans_commit.c:1138 (discriminator 1) [ +0.000028] ? __entry_text_end at ??:? [ +0.000005] bch2_trans_commit at fs/bcachefs/btree_update.h:170 (inlined by) bch2_discard_one_bucket at fs/bcachefs/alloc_background.c:1776 [ +0.000026] ? bch2_do_discards_work at fs/bcachefs/alloc_background.c:1805 (discriminator 9) [ +0.000021] bch2_do_discards_work at fs/bcachefs/alloc_background.c:1805 (discriminator 9) [ +0.000026] process_one_work at kernel/workqueue.c:3254 [ +0.000004] process_scheduled_works at kernel/workqueue.c:3329 (discriminator 2) (inlined by) worker_thread at kernel/workqueue.c:3416 (discriminator 2) [ +0.000004] ? worker_thread at kernel/workqueue.c:3362 [ +0.000002] kthread at kernel/kthread.c:388 [ +0.000003] ? kthread at kernel/kthread.c:341 [ +0.000003] ret_from_fork at arch/x86/kernel/process.c:147 [ +0.000003] ? kthread at kernel/kthread.c:341 [ +0.000002] ret_from_fork_asm at arch/x86/entry/entry_64.S:257 [ +0.000005] [ +0.000009] INFO: task bcachefs:816 blocked for more than 122 seconds. [ +0.000002] Not tainted 6.9.0-rc6-1-bcachefs-git-00178-g07f9a27f1969 #1 [ +0.000001] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ +0.000001] task:bcachefs state:D stack:0 pid:816 tgid:816 ppid:815 flags:0x00004002 [ +0.000003] Call Trace: [ +0.000001] [ +0.000002] context_switch at kernel/sched/core.c:5409 (inlined by) __schedule at kernel/sched/core.c:6746 [ +0.000003] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:182 [ +0.000004] __preempt_count_sub at arch/x86/include/asm/preempt.h:84 (discriminator 13) (inlined by) __schedule_loop at kernel/sched/core.c:6824 (discriminator 13) (inlined by) schedule at kernel/sched/core.c:6838 (discriminator 13) [ +0.000003] __closure_sync at lib/closure.c:132 (discriminator 4) [ +0.000003] closure_sync at include/linux/closure.h:194 (inlined by) bch2_btree_update_start at fs/bcachefs/btree_update_interior.c:1267 [ +0.000028] ? closure_sync_fn at lib/closure.c:112 [ +0.000003] ? _bch2_err_matches at fs/bcachefs/errcode.h:272 (inlined by) bch2_journal_replay at fs/bcachefs/recovery.c:238 [ +0.000033] bch2_btree_split_leaf at fs/bcachefs/btree_update_interior.c:1851 [ +0.000028] ? _bch2_err_matches at fs/bcachefs/errcode.h:272 (inlined by) bch2_journal_replay at fs/bcachefs/recovery.c:238 [ +0.000028] bch2_trans_commit_error at fs/bcachefs/btree_trans_commit.c:918 [ +0.000027] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:182 [ +0.000002] ? six_trylock_ip at fs/bcachefs/six.c:293 (discriminator 1) [ +0.000029] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:182 [ +0.000003] __bch2_trans_commit at fs/bcachefs/btree_trans_commit.c:1138 (discriminator 1) [ +0.000026] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:182 [ +0.000005] _bch2_err_matches at fs/bcachefs/errcode.h:272 (inlined by) bch2_journal_replay at fs/bcachefs/recovery.c:238 [ +0.000027] ? __bch2_print at fs/bcachefs/super.c:124 [ +0.000032] bch2_run_recovery_pass at fs/bcachefs/recovery_passes.c:182 [ +0.000038] bch2_run_recovery_passes at fs/bcachefs/recovery_passes.c:225 [ +0.000026] bch2_fs_recovery at fs/bcachefs/recovery.c:805 [ +0.000028] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:182 [ +0.000002] ? __bch2_sb_field_resize at fs/bcachefs/super-io.c:123 (discriminator 3) [ +0.000030] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:182 [ +0.000001] ? bch2_recalc_capacity at fs/bcachefs/alloc_background.c:2258 [ +0.000024] bch2_fs_start at fs/bcachefs/super.c:1031 [ +0.000028] bch2_fs_open at fs/bcachefs/super.c:2124 [ +0.000037] ? bch2_mount at fs/bcachefs/fs.c:1915 (discriminator 1) [ +0.000033] bch2_mount at fs/bcachefs/fs.c:1915 (discriminator 1) [ +0.000037] legacy_get_tree at fs/fs_context.c:662 [ +0.000003] vfs_get_tree at fs/super.c:1779 [ +0.000003] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:182 [ +0.000002] do_new_mount at fs/namespace.c:3352 (inlined by) path_mount at fs/namespace.c:3679 [ +0.000004] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:182 [ +0.000002] do_mount at fs/namespace.c:3693 (inlined by) __do_sys_mount at fs/namespace.c:3898 (inlined by) __se_sys_mount at fs/namespace.c:3875 (inlined by) __x64_sys_mount at fs/namespace.c:3875 [ +0.000004] do_syscall_x64 at arch/x86/entry/common.c:52 (discriminator 1) (inlined by) do_syscall_64 at arch/x86/entry/common.c:83 (discriminator 1) [ +0.000004] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:182 [ +0.000002] ? __list_del_entry_valid at include/linux/list.h:112 (inlined by) __list_del_entry at include/linux/list.h:215 (inlined by) list_del at include/linux/list.h:229 (inlined by) __remove_wait_queue at include/linux/wait.h:207 (inlined by) remove_wait_queue at kernel/sched/wait.c:55 [ +0.000002] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:182 [ +0.000001] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:182 [ +0.000002] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:182 [ +0.000001] ? n_tty_write at drivers/tty/n_tty.c:2430 (discriminator 1) [ +0.000004] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:182 [ +0.000001] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:182 [ +0.000002] ? __wake_up_common_lock at kernel/sched/wait.c:110 (inlined by) __wake_up at kernel/sched/wait.c:127 [ +0.000002] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:182 [ +0.000002] ? file_tty_write at drivers/tty/tty_io.c:1097 [ +0.000003] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:182 [ +0.000002] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:182 [ +0.000002] ? call_write_iter at include/linux/fs.h:2110 (inlined by) new_sync_write at fs/read_write.c:497 (inlined by) vfs_write at fs/read_write.c:590 [ +0.000006] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:182 [ +0.000003] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:182 [ +0.000001] ? syscall_exit_to_user_mode at kernel/entry/common.c:221 [ +0.000002] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:182 [ +0.000001] ? _static_cpu_has at arch/x86/include/asm/cpufeature.h:175 (inlined by) do_syscall_64 at arch/x86/entry/common.c:98 [ +0.000003] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:182 [ +0.000001] ? do_user_addr_fault at arch/x86/mm/fault.c:1366 [ +0.000002] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:182 [ +0.000001] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:182 [ +0.000002] entry_SYSCALL_64_after_hwframe at arch/x86/entry/entry_64.S:130 [ +0.000002] RIP: 0033:0x7f771395534e [ +0.000014] RSP: 002b:00007ffc1ae982d8 EFLAGS: 00000202 ORIG_RAX: 00000000000000a5 [ +0.000002] RAX: ffffffffffffffda RBX: 0000562c2b8a1d20 RCX: 00007f771395534e [ +0.000001] RDX: 0000562c2b881a10 RSI: 0000562c2b881b70 RDI: 0000562c2b8811e0 [ +0.000001] RBP: 00007ffc1ae986f0 R08: 0000562c2b8a1d20 R09: 0000000000000007 [ +0.000001] R10: 0000000002200000 R11: 0000000000000202 R12: 0000562c2b881b70 [ +0.000001] R13: 0000000000000009 R14: 000000000000000d R15: 0000562c2b881a10 [ +0.000003] [ +0.000003] INFO: task bch-reclaim/d60:857 blocked for more than 122 seconds. [ +0.000001] Not tainted 6.9.0-rc6-1-bcachefs-git-00178-g07f9a27f1969 #1 [ +0.000001] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ +0.000001] task:bch-reclaim/d60 state:D stack:0 pid:857 tgid:857 ppid:2 flags:0x00004000 [ +0.000002] Call Trace: [ +0.000001] [ +0.000001] context_switch at kernel/sched/core.c:5409 (inlined by) __schedule at kernel/sched/core.c:6746 [ +0.000004] __preempt_count_sub at arch/x86/include/asm/preempt.h:84 (discriminator 13) (inlined by) __schedule_loop at kernel/sched/core.c:6824 (discriminator 13) (inlined by) schedule at kernel/sched/core.c:6838 (discriminator 13) [ +0.000002] __closure_sync at lib/closure.c:132 (discriminator 4) [ +0.000003] closure_sync at include/linux/closure.h:194 (inlined by) bch2_btree_update_start at fs/bcachefs/btree_update_interior.c:1267 [ +0.000022] ? closure_sync_fn at lib/closure.c:112 [ +0.000003] ? btree_key_cache_flush_pos at fs/bcachefs/btree_key_cache.c:664 (discriminator 1) [ +0.000021] bch2_btree_split_leaf at fs/bcachefs/btree_update_interior.c:1851 [ +0.000021] ? btree_key_cache_flush_pos at fs/bcachefs/btree_key_cache.c:664 (discriminator 1) [ +0.000020] bch2_trans_commit_error at fs/bcachefs/btree_trans_commit.c:918 [ +0.000020] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:182 [ +0.000001] ? six_trylock_ip at fs/bcachefs/six.c:293 (discriminator 1) [ +0.000023] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:182 [ +0.000002] __bch2_trans_commit at fs/bcachefs/btree_trans_commit.c:1138 (discriminator 1) [ +0.000022] btree_key_cache_flush_pos at fs/bcachefs/btree_key_cache.c:664 (discriminator 1) [ +0.000024] bch2_btree_key_cache_journal_flush at fs/bcachefs/btree_key_cache.c:735 (discriminator 6) [ +0.000020] ? __btree_node_lock_nopath at fs/bcachefs/btree_locking.h:208 (inlined by) btree_node_lock_nopath_nofail at fs/bcachefs/btree_locking.h:228 (inlined by) bch2_btree_key_cache_journal_flush at fs/bcachefs/btree_key_cache.c:718 [ +0.000020] ? bch2_btree_key_cache_journal_flush at fs/bcachefs/btree_key_cache.c:709 [ +0.000020] journal_flush_pins at fs/bcachefs/journal_reclaim.c:553 [ +0.000026] __bch2_journal_reclaim at fs/bcachefs/journal_reclaim.c:685 [ +0.000023] bch2_journal_reclaim_thread at fs/bcachefs/journal_reclaim.c:728 [ +0.000021] ? bch2_journal_reclaim_thread at fs/bcachefs/journal_reclaim.c:710 [ +0.000018] kthread at kernel/kthread.c:388 [ +0.000002] ? kthread at kernel/kthread.c:341 [ +0.000002] ret_from_fork at arch/x86/kernel/process.c:147 [ +0.000002] ? kthread at kernel/kthread.c:341 [ +0.000002] ret_from_fork_asm at arch/x86/entry/entry_64.S:257 [ +0.000003] [ +0.000005] INFO: task kworker/u34:1:942 blocked for more than 122 seconds. [ +0.000001] Not tainted 6.9.0-rc6-1-bcachefs-git-00178-g07f9a27f1969 #1 [ +0.000000] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ +0.000001] task:kworker/u34:1 state:D stack:0 pid:942 tgid:942 ppid:2 flags:0x00004000 [ +0.000002] Workqueue: btree_update btree_interior_update_work [bcachefs] [ +0.000022] Call Trace: [ +0.000001] [ +0.000001] context_switch at kernel/sched/core.c:5409 (inlined by) __schedule at kernel/sched/core.c:6746 [ +0.000004] __preempt_count_sub at arch/x86/include/asm/preempt.h:84 (discriminator 13) (inlined by) __schedule_loop at kernel/sched/core.c:6824 (discriminator 13) (inlined by) schedule at kernel/sched/core.c:6838 (discriminator 13) [ +0.000002] __closure_sync at lib/closure.c:132 (discriminator 4) [ +0.000002] closure_sync at include/linux/closure.h:194 (inlined by) bch2_btree_update_start at fs/bcachefs/btree_update_interior.c:1267 [ +0.000021] ? closure_sync_fn at lib/closure.c:112 [ +0.000003] ? bch2_trans_commit at fs/bcachefs/btree_update.h:170 (inlined by) btree_update_nodes_written at fs/bcachefs/btree_update_interior.c:721 (inlined by) btree_interior_update_work at fs/bcachefs/btree_update_interior.c:859 [ +0.000020] bch2_btree_split_leaf at fs/bcachefs/btree_update_interior.c:1851 [ +0.000020] ? bch2_trans_commit at fs/bcachefs/btree_update.h:170 (inlined by) btree_update_nodes_written at fs/bcachefs/btree_update_interior.c:721 (inlined by) btree_interior_update_work at fs/bcachefs/btree_update_interior.c:859 [ +0.000019] bch2_trans_commit_error at fs/bcachefs/btree_trans_commit.c:918 [ +0.000020] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:182 [ +0.000001] ? six_trylock_ip at fs/bcachefs/six.c:293 (discriminator 1) [ +0.000022] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:182 [ +0.000002] __bch2_trans_commit at fs/bcachefs/btree_trans_commit.c:1138 (discriminator 1) [ +0.000022] bch2_trans_commit at fs/bcachefs/btree_update.h:170 (inlined by) btree_update_nodes_written at fs/bcachefs/btree_update_interior.c:721 (inlined by) btree_interior_update_work at fs/bcachefs/btree_update_interior.c:859 [ +0.000022] ? process_one_work at kernel/workqueue.c:3254 [ +0.000002] ? srso_alias_return_thunk at arch/x86/lib/retpoline.S:182 [ +0.000002] process_one_work at kernel/workqueue.c:3254 [ +0.000002] process_scheduled_works at kernel/workqueue.c:3329 (discriminator 2) (inlined by) worker_thread at kernel/workqueue.c:3416 (discriminator 2) [ +0.000003] ? worker_thread at kernel/workqueue.c:3362 [ +0.000002] kthread at kernel/kthread.c:388 [ +0.000001] ? kthread at kernel/kthread.c:341 [ +0.000002] ret_from_fork at arch/x86/kernel/process.c:147 [ +0.000002] ? kthread at kernel/kthread.c:341 [ +0.000002] ret_from_fork_asm at arch/x86/entry/entry_64.S:257 [ +0.000003] ```

Just one question, __entry_text_end returned me ??:?, is this normal? do i have something worng with my configs?

Valmar33 commented 2 months ago

Sorry for the delay, here it is. addessed dmesg

Just one question, __entry_text_end returned me ??:?, is this normal? do i have something worng with my configs?

You can always search for the string in the Github repo:

https://github.com/search?q=repo%3Akoverstreet%2Fbcachefs+__entry_text_end&type=code

Basically, just looks like some assembly that the decoder doesn't know how to handle. Doesn't look relevant to anything, anyways.

Snogard commented 2 months ago

Sorry for the delay, here it is. addessed dmesg Just one question, __entry_text_end returned me ??:?, is this normal? do i have something worng with my configs?

You can always search for the string in the Github repo:

https://github.com/search?q=repo%3Akoverstreet%2Fbcachefs+__entry_text_end&type=code

Basically, just looks like some assembly that the decoder doesn't know how to handle. Doesn't look relevant to anything, anyways.

thanks for the clarification!

Snogard commented 1 month ago

Any news on this?

Snogard commented 3 weeks ago

Should i try again with that commit or shoud i wait a bit more?