Cannot mount filesystem after unclean shutdown (kernel thread hangs during journal replay)

[inodentry]

I have an Arch Linux system that is installed to bcachefs. It is booting using EFISTUB with the kernel on a small EFI partition, everything else is bcachefs. It is a "home media server" running some Docker containers, Kodi, a few other things...

Just to give some relevant info about things I have done in the configuration:

It has two SSDs (/dev/sda1, /dev/sdd1) for foreground+promote and two HDDs (/dev/sdb, /dev/sdc) for background. Replicas=2 at the FS level. Replicas=1 set selectively on some directories. xxhash checksums, no compression.

At one point, I decided that I don't want the rootfs to go on the background devices anymore, so I did bcachefs setattr --background_target=ssd / (set the background to ssd on the root of the filesystem), and then set it back on specific directories like bcachefs setattr --background_target=hdd /srv. I am not sure if this is a good idea, but oh well, it seemed to work.

Kernel: 6.7.6-arch1-2

The issue is that I can no longer mount my filesystem, because it hangs during mount during journal replay. I am now trying to mount it from a live ISO to at least collect dmesg output.

Relevant dmesg output:

[    0.719587] sd 2:0:0:0: [sda] 976773168 512-byte logical blocks: (500 GB/466 GiB)
[    0.719589] sd 2:0:0:0: [sda] 4096-byte physical blocks
[    0.719593] sd 2:0:0:0: [sda] Write Protect is off
[    0.719594] sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00
[    0.719600] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    0.719612] sd 2:0:0:0: [sda] Preferred minimum I/O size 4096 bytes
[    0.719657] scsi 3:0:0:0: Direct-Access     ATA      HGST HTS721010A9 A3J0 PQ: 0 ANSI: 5
[    0.719821] sd 3:0:0:0: [sdb] 1953525168 512-byte logical blocks: (1.00 TB/932 GiB)
[    0.719822] sd 3:0:0:0: [sdb] 4096-byte physical blocks
[    0.719827] sd 3:0:0:0: [sdb] Write Protect is off
[    0.719828] sd 3:0:0:0: [sdb] Mode Sense: 00 3a 00 00
[    0.719836] sd 3:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    0.719844] sd 3:0:0:0: [sdb] Preferred minimum I/O size 4096 bytes
[    0.719845] scsi 4:0:0:0: Direct-Access     ATA      HGST HTS721010A9 A3V0 PQ: 0 ANSI: 5
[    0.719959] sd 4:0:0:0: [sdc] 1953525168 512-byte logical blocks: (1.00 TB/932 GiB)
[    0.719961] sd 4:0:0:0: [sdc] 4096-byte physical blocks
[    0.719965] sd 4:0:0:0: [sdc] Write Protect is off
[    0.719966] sd 4:0:0:0: [sdc] Mode Sense: 00 3a 00 00
[    0.719974] sd 4:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    0.719988] sd 4:0:0:0: [sdc] Preferred minimum I/O size 4096 bytes
[    0.719996] scsi 5:0:0:0: Direct-Access     ATA      R3SL480G         2C   PQ: 0 ANSI: 5
[    0.720135] sd 5:0:0:0: [sdd] 937703088 512-byte logical blocks: (480 GB/447 GiB)
[    0.720139] sd 5:0:0:0: [sdd] Write Protect is off
[    0.720140] sd 5:0:0:0: [sdd] Mode Sense: 00 3a 00 00
[    0.720146] sd 5:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    0.720154] sd 5:0:0:0: [sdd] Preferred minimum I/O size 512 bytes
[    0.721044]  sdd: sdd1
[    0.721082] sd 5:0:0:0: [sdd] Attached SCSI disk
[    0.721237]  sda: sda1 sda2 sda3
[    0.721392] sd 2:0:0:0: [sda] supports TCG Opal
[    0.721393] sd 2:0:0:0: [sda] Attached SCSI disk
[    0.987275] sd 3:0:0:0: [sdb] Attached SCSI disk
[    1.003645] sd 4:0:0:0: [sdc] Attached SCSI disk

[  297.162442] bcachefs (0cda7e0c-e826-4f4b-a499-d93bb20ed966): mounting version 1.3: rebalance_work opts=metadata_replicas=2,data_replicas=2,metadata_checksum=xxhash,data_checksum=xxhash,metadata_target=ssd,foreground_target=ssd,background_target=hdd,promote_target=ssd,root_reserve_percent=2
[  297.162446] bcachefs (0cda7e0c-e826-4f4b-a499-d93bb20ed966): recovering from unclean shutdown
[  303.801614] bcachefs (0cda7e0c-e826-4f4b-a499-d93bb20ed966): journal read done, replaying entries 439092-440516
[  304.845607] bcachefs (0cda7e0c-e826-4f4b-a499-d93bb20ed966): alloc_read... done
[  304.867014] bcachefs (0cda7e0c-e826-4f4b-a499-d93bb20ed966): stripes_read... done
[  304.867020] bcachefs (0cda7e0c-e826-4f4b-a499-d93bb20ed966): snapshots_read... done
[  304.994570] bcachefs (0cda7e0c-e826-4f4b-a499-d93bb20ed966): journal_replay...
[  305.061079] bcachefs (0cda7e0c-e826-4f4b-a499-d93bb20ed966): going read-write

[  859.964406] INFO: task kworker/u16:1:13 blocked for more than 491 seconds.
[  859.964473]       Tainted: P           OE      6.7.2-arch1-2 #1
[  859.964510] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  859.964553] task:kworker/u16:1   state:D stack:0     pid:13    tgid:13    ppid:2      flags:0x00004000
[  859.964569] Workqueue: btree_update btree_interior_update_work [bcachefs]
[  859.964827] Call Trace:
[  859.964831]  <TASK>
[  859.964840]  __schedule+0x3e7/0x1410
[  859.964858]  ? __bch2_time_stats_update+0x11c/0x270 [bcachefs b6b77aaf0e90cfaebdb766a3d082fa79e52144ab]
[  859.965120]  schedule+0x32/0xd0
[  859.965129]  __closure_sync+0x82/0x160
[  859.965145]  bch2_btree_update_start+0x928/0x940 [bcachefs b6b77aaf0e90cfaebdb766a3d082fa79e52144ab]
[  859.965394]  ? __pfx_closure_sync_fn+0x10/0x10
[  859.965408]  ? __bch2_foreground_maybe_merge+0x512/0xd10 [bcachefs b6b77aaf0e90cfaebdb766a3d082fa79e52144ab]
[  859.965644]  __bch2_foreground_maybe_merge+0x512/0xd10 [bcachefs b6b77aaf0e90cfaebdb766a3d082fa79e52144ab]
[  859.965892]  ? run_btree_triggers+0x205/0x3b0 [bcachefs b6b77aaf0e90cfaebdb766a3d082fa79e52144ab]
[  859.966130]  __bch2_trans_commit+0xcf8/0x1820 [bcachefs b6b77aaf0e90cfaebdb766a3d082fa79e52144ab]
[  859.966372]  btree_interior_update_work+0x28f/0x9e0 [bcachefs b6b77aaf0e90cfaebdb766a3d082fa79e52144ab]
[  859.966607]  ? btree_interior_update_work+0x106/0x9e0 [bcachefs b6b77aaf0e90cfaebdb766a3d082fa79e52144ab]
[  859.966845]  process_one_work+0x171/0x340
[  859.966857]  worker_thread+0x27b/0x3a0
[  859.966868]  ? __pfx_worker_thread+0x10/0x10
[  859.966876]  kthread+0xe5/0x120
[  859.966887]  ? __pfx_kthread+0x10/0x10
[  859.966898]  ret_from_fork+0x31/0x50
[  859.966908]  ? __pfx_kthread+0x10/0x10
[  859.966918]  ret_from_fork_asm+0x1b/0x30
[  859.966933]  </TASK>
[  859.966937] Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings

bcachefs show-super output:

External UUID:                              0cda7e0c-e826-4f4b-a499-d93bb20ed966
Internal UUID:                              88a840a2-4595-4b47-a58a-d9ebf955800d
Magic number:                               c68573f6-66ce-90a9-d96a-60cf803df7ef
Device index:                               0
Label:
Version:                                    1.3: rebalance_work
Version upgrade complete:                   1.3: rebalance_work
Oldest version on disk:                     1.3: rebalance_work
Created:                                    Sat Feb 24 18:45:40 2024
Sequence number:                            46
Time of last write:                         Sat Feb 24 18:45:41 2024
Superblock size:                            6472
Clean:                                      0
Devices:                                    4
Sections:                                   members_v1,replicas_v0,disk_groups,clean,journal_seq_blacklist,journal_v2,counters,members_v2,errors,ext,downgrade
Features:                                   journal_seq_blacklist_v3,reflink,new_siphash,inline_data,new_extent_overwrite,btree_ptr_v2,extents_above_btree_updates,btree_updates_journalled,reflink_inline_data,new_varint,journal_no_flush,alloc_v2,extents_across_btree_nodes
Compat features:                            alloc_info,alloc_metadata,extents_above_btree_updates_done,bformat_overflow_done

Options:
  block_size:                               4.00 KiB
  btree_node_size:                          256 KiB
  errors:                                   continue [ro] panic
  metadata_replicas:                        2
  data_replicas:                            2
  metadata_replicas_required:               1
  data_replicas_required:                   1
  encoded_extent_max:                       64.0 KiB
  metadata_checksum:                        none crc32c crc64 [xxhash]
  data_checksum:                            none crc32c crc64 [xxhash]
  compression:                              none
  background_compression:                   none
  str_hash:                                 crc32c crc64 [siphash]
  metadata_target:                          ssd
  foreground_target:                        ssd
  background_target:                        hdd
  promote_target:                           ssd
  erasure_code:                             0
  inodes_32bit:                             1
  shard_inode_numbers:                      1
  inodes_use_key_cache:                     1
  gc_reserve_percent:                       8
  gc_reserve_bytes:                         0 B
  root_reserve_percent:                     2
  wide_macs:                                0
  acl:                                      1
  usrquota:                                 0
  grpquota:                                 0
  prjquota:                                 0
  journal_flush_delay:                      1000
  journal_flush_disabled:                   0
  journal_reclaim_delay:                    100
  journal_transaction_names:                1
  version_upgrade:                          [compatible] incompatible none
  nocow:                                    0

members_v2 (size 528):
Device:                                     0
  Label:                                    ssd1 (1)
  UUID:                                     79ecf337-747a-4bb8-981b-34c92a422735
  Size:                                     447 GiB
  read errors:                              0
  write errors:                             0
  checksum errors:                          0
  seqread iops:                             0
  seqwrite iops:                            0
  randread iops:                            0
  randwrite iops:                           0
  Bucket size:                              256 KiB
  First bucket:                             0
  Buckets:                                  1831444
  Last mount:                               Thu Feb 29 15:33:32 2024
  Last superblock write:                    7
  State:                                    rw
  Data allowed:                             journal,btree,user
  Has data:                                 journal,btree,user,cached
  Durability:                               1
  Discard:                                  0
  Freespace initialized:                    1
Device:                                     1
  Label:                                    ssd2 (2)
  UUID:                                     8cb0e37c-3d83-4237-92c2-6aa6424f9cc8
  Size:                                     447 GiB
  read errors:                              0
  write errors:                             0
  checksum errors:                          0
  seqread iops:                             0
  seqwrite iops:                            0
  randread iops:                            0
  randwrite iops:                           0
  Bucket size:                              256 KiB
  First bucket:                             0
  Buckets:                                  1831444
  Last mount:                               Thu Feb 29 15:33:32 2024
  Last superblock write:                    7
  State:                                    rw
  Data allowed:                             journal,btree,user
  Has data:                                 journal,btree,user,cached
  Durability:                               1
  Discard:                                  0
  Freespace initialized:                    1
Device:                                     2
  Label:                                    hdd1 (4)
  UUID:                                     bbe1cb36-ac33-4518-81b9-5d635326cf37
  Size:                                     932 GiB
  read errors:                              0
  write errors:                             0
  checksum errors:                          0
  seqread iops:                             0
  seqwrite iops:                            0
  randread iops:                            0
  randwrite iops:                           0
  Bucket size:                              256 KiB
  First bucket:                             0
  Buckets:                                  3815478
  Last mount:                               Thu Feb 29 15:33:32 2024
  Last superblock write:                    7
  State:                                    rw
  Data allowed:                             journal,btree,user
  Has data:                                 user,cached
  Durability:                               1
  Discard:                                  0
  Freespace initialized:                    1
Device:                                     3
  Label:                                    hdd2 (5)
  UUID:                                     6b984599-4c1b-4c0c-b387-0b89726e2871
  Size:                                     932 GiB
  read errors:                              0
  write errors:                             0
  checksum errors:                          0
  seqread iops:                             0
  seqwrite iops:                            0
  randread iops:                            0
  randwrite iops:                           0
  Bucket size:                              256 KiB
  First bucket:                             0
  Buckets:                                  3815478
  Last mount:                               Thu Feb 29 15:33:32 2024
  Last superblock write:                    7
  State:                                    rw
  Data allowed:                             journal,btree,user
  Has data:                                 user,cached
  Durability:                               1
  Discard:                                  0
  Freespace initialized:                    1

errors (size 8):

Running bcachefs fsck -v /dev/sda1:/dev/sdb:/dev/sdc:/dev/sdd1:

mounting version 1.3: rebalance_work opts=ro,metadata_replicas=2,data_replicas=2,metadata_checksum=xxhash,data_checksum=xxhash,metadata_target=ssd,foreground_target=ssd,background_target=hdd,promote_target=ssd,root_reserve_percent=2,degraded,verbose,fsck,fix_errors=ask,read_only
recovering from unclean shutdown
Doing compatible version upgrade from 1.3: rebalance_work to 1.4: member_seq

starting journal read
journal read done on device /dev/sda1, ret 0
journal read done on device /dev/sdd1, ret 0
journal read done on device /dev/sdb, ret 0
journal read done on device /dev/sdc, ret 0
journal read done, replaying entries 439092-440606
Journal keys: 893109 read, 234741 after sorting and compacting
alloc_read... done
stripes_read... done
snapshots_read... done
check_allocations... done
going read-write
journal_replay...bch2_btree_update_start(): error journal_reclaim_would_deadlock
bch2_btree_update_start(): error journal_reclaim_would_deadlock
bch2_btree_update_start(): error journal_reclaim_would_deadlock
bch2_btree_update_start(): error journal_reclaim_would_deadlock
bch2_btree_update_start(): error journal_reclaim_would_deadlock
bch2_btree_update_start(): error journal_reclaim_would_deadlock
bch2_btree_update_start(): error journal_reclaim_would_deadlock
bch2_btree_update_start(): error journal_reclaim_would_deadlock
bch2_btree_update_start(): error journal_reclaim_would_deadlock
bch2_btree_update_start(): error journal_reclaim_would_deadlock
bch2_btree_update_start: 18123 callbacks suppressed
bch2_btree_update_start(): error journal_reclaim_would_deadlock
...

Please let me know what I could do, if anything, to try to safely recover my filesystem?

Or if you'd like anything else from me to help you diagnose and fix this bug?

[ramonacat]

Hey, so I've run into this issue again. I'm running the latest master from this repository (commit c5fc668464fce66952294d681bf5b2944f666122), but unfortunately my filesystem doesn't mount:

[  513.015627] bcachefs (73a74159-dba2-49c1-b386-04ab78af84e3): mounting version 1.4: member_seq opts=metadata_replicas=2,data_replicas=2
[  513.015633] bcachefs (73a74159-dba2-49c1-b386-04ab78af84e3): recovering from unclean shutdown
[  513.015635] bcachefs (73a74159-dba2-49c1-b386-04ab78af84e3): superblock requires following recovery passes to be run:
                 check_dirents
[  513.015640] bcachefs (73a74159-dba2-49c1-b386-04ab78af84e3): Doing compatible version upgrade from 1.4: member_seq to 1.7: mi_btree_bitmap
                 running recovery passes: check_allocations,check_subvols
[  513.986477] bcachefs (73a74159-dba2-49c1-b386-04ab78af84e3): journal read done, replaying entries 3934173-3936073
[  514.694753] bcachefs (73a74159-dba2-49c1-b386-04ab78af84e3): alloc_read... done
[  514.700638] bcachefs (73a74159-dba2-49c1-b386-04ab78af84e3): stripes_read... done
[  514.700643] bcachefs (73a74159-dba2-49c1-b386-04ab78af84e3): snapshots_read... done
[  514.702264] bcachefs (73a74159-dba2-49c1-b386-04ab78af84e3): check_allocations... done
[  566.228127] bcachefs (73a74159-dba2-49c1-b386-04ab78af84e3): going read-write
[  566.228326] bcachefs (73a74159-dba2-49c1-b386-04ab78af84e3): journal_replay... done
[  567.147822] bcachefs (73a74159-dba2-49c1-b386-04ab78af84e3): check_subvols... done
[  567.151595] bcachefs (73a74159-dba2-49c1-b386-04ab78af84e3): check_dirents...
[  567.151696] dirent points to missing inode:
[  567.151698] u64s 8 type dirent 4097:6316666529397969:U32_MAX len 0 ver 0: 537000643 -> 537000643 type reg, shutting down
[  567.151734] bcachefs (73a74159-dba2-49c1-b386-04ab78af84e3): inconsistency detected - emergency read only at journal seq 3936089
[  567.151754] bcachefs (73a74159-dba2-49c1-b386-04ab78af84e3): check_dirent(): error fsck_errors_not_fixed
[  567.151773] bcachefs (73a74159-dba2-49c1-b386-04ab78af84e3): bch2_check_dirents(): error fsck_errors_not_fixed
[  567.153625] bcachefs (73a74159-dba2-49c1-b386-04ab78af84e3): bch2_fs_recovery(): error fsck_errors_not_fixed
[  567.153643] bcachefs (73a74159-dba2-49c1-b386-04ab78af84e3): bch2_fs_start(): error starting filesystem fsck_errors_not_fixed
[  567.159595] bcachefs (73a74159-dba2-49c1-b386-04ab78af84e3): unshutdown complete, journal seq 3936089

I won't reformat this time, unless that's the only way to recover :)

[koverstreet]

Mounting with -o fsck,fix_errors should fix this. If you're up for sending me a metadata dump, and the transaction that caused this is still in the journal, I might be able to track down the actual bug.

[ramonacat]

Sorry, I wasn't able to get a metadata dump. But I'm confirming that the filesystem mounted after the fsck, and seems to work correctly now :partying_face:. Thank you for all your great work Kent!

[PrefersAwkward]

I didn't have an opportunity to test for the last week BUT I just used master a few minutes ago, and it recovered my FS.

Amazing work!

[cotsuka]

Was able to spend the last few hours building a kernel with master branch and confirming that mounting with fsck,fix_errors,no_splitbrain_check seems to have fixed my filesystem. I originally added the flags to my fstab which it didn't seem to like, but running mount directly worked. Took a few hours to finish.

[Manoa1911]

when I run -o fsck,fix_errors,no_splitbrain_check the mount command freezed and no error messages bcfs.txt

[koverstreet]

that's not much to go on. was it spinning, using cpu?

[Manoa1911]

yes, was, until the end of this dmesg, it was doing something, this is very similar to other runs I did with various other parameters like ro/noupgrade/very_degraded/nochanges..... nothing worked yet :(

and I can't run anything on the devices after this command it says the device is busy when I try a normal mount

[koverstreet]

check what it's doing with perf top

On Mon, May 27, 2024 at 9:15 PM dod @.***> wrote:

yes, was, until the end of this dmesg, it was doing something, this is very similar to other runs I did with various other parameters like ro/noupgrade/very_degraded/nochanges..... nothing worked yet :(

— Reply to this email directly, view it on GitHub https://github.com/koverstreet/bcachefs/issues/656#issuecomment-2134194742, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAPGX3XBGY6XX747UMFQXBTZEPLENAVCNFSM6AAAAABD77II5KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZUGE4TINZUGI . You are receiving this because you were mentioned.Message ID: @.***>

[Manoa1911]

this it's: perftop.txt

thank for help :) ....any suggestions on what to do next ? ...I downloaded tools from 4 hours ago commit 1501, im trying to resurrection the file system :)

it freezed on this:

invalid bkey on insert from bch2_journal_replay -> 0x559523418b1es
  u64s 12 type btree_ptr_v2 92442:1992475006015864008:U32_MAX len 0 ver 0: seq 4388538dc1a80624 written 0 min_key R 68883:1899175029502497872:0 durability: 2 ptr: 2:1967860:0 gen 10 ptr: 1:2052096:0 gen 9
  sectors_written == 0
transaction updates for bch2_journal_replay journal seq 4294967295
  update: btree=dirents cached=0 0x559523418b1eS
    old u64s 12 type btree_ptr_v2 92442:1992475006015864008:U32_MAX len 0 ver 0: seq 4388538dc1a80624 written 0 min_key R 68883:1899175029502497872:0 durability: 2 ptr: 2:1967860:0 gen 10 ptr: 1:2052096:0 gen 9
    new u64s 12 type btree_ptr_v2 92442:1992475006015864008:U32_MAX len 0 ver 0: seq 4388538dc1a80624 written 0 min_key R 68883:1899175029502497872:0 durability: 2 ptr: 2:1967860:0 gen 10 ptr: 1:2052096:0 gen 9

invalid bkey on insert from bch2_journal_replay -> 0x559523418b1es
  u64s 12 type btree_ptr_v2 114718:8808187908021241809:U32_MAX len 0 ver 0: seq c93dc8da67b6b2c8 written 0 min_key R 109932:4788216074792127587:0 durability: 2 ptr: 2:2165605:512 gen 3 ptr: 3:2174882:0 gen 3
  sectors_written == 0
transaction updates for bch2_journal_replay journal seq 4294967295
  update: btree=dirents cached=0 0x559523418b1eS
    old u64s 12 type btree_ptr_v2 114718:8808187908021241809:U32_MAX len 0 ver 0: seq c93dc8da67b6b2c8 written 0 min_key R 109932:4788216074792127587:0 durability: 2 ptr: 2:2165605:512 gen 3 ptr: 3:2174882:0 gen 3
    new u64s 12 type btree_ptr_v2 114718:8808187908021241809:U32_MAX len 0 ver 0: seq c93dc8da67b6b2c8 written 0 min_key R 109932:4788216074792127587:0 durability: 2 ptr: 2:2165605:512 gen 3 ptr: 3:2174882:0 gen 3

invalid bkey on insert from bch2_journal_replay -> 0x559523418b1es
  u64s 11 type btree_ptr_v2 170216:3259457490083641562:U32_MAX len 0 ver 0: seq e87d6c298993113f written 0 min_key R 162720:6545793145879759886:0 durability: 1 ptr: 1:3324723:512 gen 12
  sectors_written == 0
transaction updates for bch2_journal_replay journal seq 4294967295
  update: btree=dirents cached=0 0x559523418b1eS
    old u64s 11 type btree_ptr_v2 170216:3259457490083641562:U32_MAX len 0 ver 0: seq e87d6c298993113f written 0 min_key R 162720:6545793145879759886:0 durability: 1 ptr: 1:3324723:512 gen 12
    new u64s 11 type btree_ptr_v2 170216:3259457490083641562:U32_MAX len 0 ver 0: seq e87d6c298993113f written 0 min_key R 162720:6545793145879759886:0 durability: 1 ptr: 1:3324723:512 gen 12

invalid bkey on insert from bch2_journal_replay -> 0x559523418b1es
  u64s 12 type btree_ptr_v2 346738:7053484335289139072:U32_MAX len 0 ver 0: seq 5b30e2674b39a498 written 0 min_key R 235088:3216720161223041375:0 durability: 2 ptr: 3:1896528:0 gen 11 ptr: 2:3164264:0 gen 10
  sectors_written == 0
transaction updates for bch2_journal_replay journal seq 4294967295
  update: btree=dirents cached=0 0x559523418b1eS
    old u64s 12 type btree_ptr_v2 346738:7053484335289139072:U32_MAX len 0 ver 0: seq 5b30e2674b39a498 written 0 min_key R 235088:3216720161223041375:0 durability: 2 ptr: 3:1896528:0 gen 11 ptr: 2:3164264:0 gen 10
    new u64s 12 type btree_ptr_v2 346738:7053484335289139072:U32_MAX len 0 ver 0: seq 5b30e2674b39a498 written 0 min_key R 235088:3216720161223041375:0 durability: 2 ptr: 3:1896528:0 gen 11 ptr: 2:3164264:0 gen 10

invalid bkey on insert from bch2_journal_replay -> 0x559523418b1es
  u64s 11 type btree_ptr_v2 SPOS_MAX len 0 ver 0: seq a20e7ee3a59548b6 written 0 min_key R 346738:7053484335289139073:0 durability: 1 ptr: 1:2166957:512 gen 3
  sectors_written == 0
transaction updates for bch2_journal_replay journal seq 4294967295
  update: btree=dirents cached=0 0x559523418b1eS
    old u64s 11 type btree_ptr_v2 SPOS_MAX len 0 ver 0: seq a20e7ee3a59548b6 written 0 min_key R 346738:7053484335289139073:0 durability: 1 ptr: 1:2166957:512 gen 3
    new u64s 11 type btree_ptr_v2 SPOS_MAX len 0 ver 0: seq a20e7ee3a59548b6 written 0 min_key R 346738:7053484335289139073:0 durability: 1 ptr: 1:2166957:512 gen 3

bch2_btree_update_start: 237 callbacks suppressed
bch2_btree_update_start(): error Input/output error
bch2_btree_update_start(): error Input/output error
bch2_btree_update_start(): error Input/output error
bch2_journal_replay(): error while replaying key at btree inodes level 0: Input/output error
bch2_journal_replay(): error Input/output error

or mybe it still running, I see on htop 0.5% every now and then.. no errors on dmesg at all ...it was a prompt but it didn't write it, I pressed enter and it asked Y/N it looks like it get to the end and say unshutdown complete but not yet released the console and no activity in htop shrinkers thread is running once every 5 seconds for up to 1.5% and that's it memory use is dropping by around 1 MB every 2 hours or so, I guess it's doing something