Open inodentry opened 6 months ago
Hey, so I've run into this issue again. I'm running the latest master from this repository (commit c5fc668464fce66952294d681bf5b2944f666122), but unfortunately my filesystem doesn't mount:
[ 513.015627] bcachefs (73a74159-dba2-49c1-b386-04ab78af84e3): mounting version 1.4: member_seq opts=metadata_replicas=2,data_replicas=2
[ 513.015633] bcachefs (73a74159-dba2-49c1-b386-04ab78af84e3): recovering from unclean shutdown
[ 513.015635] bcachefs (73a74159-dba2-49c1-b386-04ab78af84e3): superblock requires following recovery passes to be run:
check_dirents
[ 513.015640] bcachefs (73a74159-dba2-49c1-b386-04ab78af84e3): Doing compatible version upgrade from 1.4: member_seq to 1.7: mi_btree_bitmap
running recovery passes: check_allocations,check_subvols
[ 513.986477] bcachefs (73a74159-dba2-49c1-b386-04ab78af84e3): journal read done, replaying entries 3934173-3936073
[ 514.694753] bcachefs (73a74159-dba2-49c1-b386-04ab78af84e3): alloc_read... done
[ 514.700638] bcachefs (73a74159-dba2-49c1-b386-04ab78af84e3): stripes_read... done
[ 514.700643] bcachefs (73a74159-dba2-49c1-b386-04ab78af84e3): snapshots_read... done
[ 514.702264] bcachefs (73a74159-dba2-49c1-b386-04ab78af84e3): check_allocations... done
[ 566.228127] bcachefs (73a74159-dba2-49c1-b386-04ab78af84e3): going read-write
[ 566.228326] bcachefs (73a74159-dba2-49c1-b386-04ab78af84e3): journal_replay... done
[ 567.147822] bcachefs (73a74159-dba2-49c1-b386-04ab78af84e3): check_subvols... done
[ 567.151595] bcachefs (73a74159-dba2-49c1-b386-04ab78af84e3): check_dirents...
[ 567.151696] dirent points to missing inode:
[ 567.151698] u64s 8 type dirent 4097:6316666529397969:U32_MAX len 0 ver 0: 537000643 -> 537000643 type reg, shutting down
[ 567.151734] bcachefs (73a74159-dba2-49c1-b386-04ab78af84e3): inconsistency detected - emergency read only at journal seq 3936089
[ 567.151754] bcachefs (73a74159-dba2-49c1-b386-04ab78af84e3): check_dirent(): error fsck_errors_not_fixed
[ 567.151773] bcachefs (73a74159-dba2-49c1-b386-04ab78af84e3): bch2_check_dirents(): error fsck_errors_not_fixed
[ 567.153625] bcachefs (73a74159-dba2-49c1-b386-04ab78af84e3): bch2_fs_recovery(): error fsck_errors_not_fixed
[ 567.153643] bcachefs (73a74159-dba2-49c1-b386-04ab78af84e3): bch2_fs_start(): error starting filesystem fsck_errors_not_fixed
[ 567.159595] bcachefs (73a74159-dba2-49c1-b386-04ab78af84e3): unshutdown complete, journal seq 3936089
I won't reformat this time, unless that's the only way to recover :)
Mounting with -o fsck,fix_errors should fix this. If you're up for sending me a metadata dump, and the transaction that caused this is still in the journal, I might be able to track down the actual bug.
Sorry, I wasn't able to get a metadata dump. But I'm confirming that the filesystem mounted after the fsck, and seems to work correctly now :partying_face:. Thank you for all your great work Kent!
I didn't have an opportunity to test for the last week BUT I just used master a few minutes ago, and it recovered my FS.
Amazing work!
Was able to spend the last few hours building a kernel with master
branch and confirming that mounting with fsck,fix_errors,no_splitbrain_check
seems to have fixed my filesystem. I originally added the flags to my fstab which it didn't seem to like, but running mount
directly worked. Took a few hours to finish.
when I run -o fsck,fix_errors,no_splitbrain_check the mount command freezed and no error messages bcfs.txt
that's not much to go on. was it spinning, using cpu?
yes, was, until the end of this dmesg, it was doing something, this is very similar to other runs I did with various other parameters like ro/noupgrade/very_degraded/nochanges..... nothing worked yet :(
and I can't run anything on the devices after this command it says the device is busy when I try a normal mount
check what it's doing with perf top
On Mon, May 27, 2024 at 9:15 PM dod @.***> wrote:
yes, was, until the end of this dmesg, it was doing something, this is very similar to other runs I did with various other parameters like ro/noupgrade/very_degraded/nochanges..... nothing worked yet :(
— Reply to this email directly, view it on GitHub https://github.com/koverstreet/bcachefs/issues/656#issuecomment-2134194742, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAPGX3XBGY6XX747UMFQXBTZEPLENAVCNFSM6AAAAABD77II5KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZUGE4TINZUGI . You are receiving this because you were mentioned.Message ID: @.***>
this it's: perftop.txt
thank for help :) ....any suggestions on what to do next ? ...I downloaded tools from 4 hours ago commit 1501, im trying to resurrection the file system :)
it freezed on this:
invalid bkey on insert from bch2_journal_replay -> 0x559523418b1es
u64s 12 type btree_ptr_v2 92442:1992475006015864008:U32_MAX len 0 ver 0: seq 4388538dc1a80624 written 0 min_key R 68883:1899175029502497872:0 durability: 2 ptr: 2:1967860:0 gen 10 ptr: 1:2052096:0 gen 9
sectors_written == 0
transaction updates for bch2_journal_replay journal seq 4294967295
update: btree=dirents cached=0 0x559523418b1eS
old u64s 12 type btree_ptr_v2 92442:1992475006015864008:U32_MAX len 0 ver 0: seq 4388538dc1a80624 written 0 min_key R 68883:1899175029502497872:0 durability: 2 ptr: 2:1967860:0 gen 10 ptr: 1:2052096:0 gen 9
new u64s 12 type btree_ptr_v2 92442:1992475006015864008:U32_MAX len 0 ver 0: seq 4388538dc1a80624 written 0 min_key R 68883:1899175029502497872:0 durability: 2 ptr: 2:1967860:0 gen 10 ptr: 1:2052096:0 gen 9
invalid bkey on insert from bch2_journal_replay -> 0x559523418b1es
u64s 12 type btree_ptr_v2 114718:8808187908021241809:U32_MAX len 0 ver 0: seq c93dc8da67b6b2c8 written 0 min_key R 109932:4788216074792127587:0 durability: 2 ptr: 2:2165605:512 gen 3 ptr: 3:2174882:0 gen 3
sectors_written == 0
transaction updates for bch2_journal_replay journal seq 4294967295
update: btree=dirents cached=0 0x559523418b1eS
old u64s 12 type btree_ptr_v2 114718:8808187908021241809:U32_MAX len 0 ver 0: seq c93dc8da67b6b2c8 written 0 min_key R 109932:4788216074792127587:0 durability: 2 ptr: 2:2165605:512 gen 3 ptr: 3:2174882:0 gen 3
new u64s 12 type btree_ptr_v2 114718:8808187908021241809:U32_MAX len 0 ver 0: seq c93dc8da67b6b2c8 written 0 min_key R 109932:4788216074792127587:0 durability: 2 ptr: 2:2165605:512 gen 3 ptr: 3:2174882:0 gen 3
invalid bkey on insert from bch2_journal_replay -> 0x559523418b1es
u64s 11 type btree_ptr_v2 170216:3259457490083641562:U32_MAX len 0 ver 0: seq e87d6c298993113f written 0 min_key R 162720:6545793145879759886:0 durability: 1 ptr: 1:3324723:512 gen 12
sectors_written == 0
transaction updates for bch2_journal_replay journal seq 4294967295
update: btree=dirents cached=0 0x559523418b1eS
old u64s 11 type btree_ptr_v2 170216:3259457490083641562:U32_MAX len 0 ver 0: seq e87d6c298993113f written 0 min_key R 162720:6545793145879759886:0 durability: 1 ptr: 1:3324723:512 gen 12
new u64s 11 type btree_ptr_v2 170216:3259457490083641562:U32_MAX len 0 ver 0: seq e87d6c298993113f written 0 min_key R 162720:6545793145879759886:0 durability: 1 ptr: 1:3324723:512 gen 12
invalid bkey on insert from bch2_journal_replay -> 0x559523418b1es
u64s 12 type btree_ptr_v2 346738:7053484335289139072:U32_MAX len 0 ver 0: seq 5b30e2674b39a498 written 0 min_key R 235088:3216720161223041375:0 durability: 2 ptr: 3:1896528:0 gen 11 ptr: 2:3164264:0 gen 10
sectors_written == 0
transaction updates for bch2_journal_replay journal seq 4294967295
update: btree=dirents cached=0 0x559523418b1eS
old u64s 12 type btree_ptr_v2 346738:7053484335289139072:U32_MAX len 0 ver 0: seq 5b30e2674b39a498 written 0 min_key R 235088:3216720161223041375:0 durability: 2 ptr: 3:1896528:0 gen 11 ptr: 2:3164264:0 gen 10
new u64s 12 type btree_ptr_v2 346738:7053484335289139072:U32_MAX len 0 ver 0: seq 5b30e2674b39a498 written 0 min_key R 235088:3216720161223041375:0 durability: 2 ptr: 3:1896528:0 gen 11 ptr: 2:3164264:0 gen 10
invalid bkey on insert from bch2_journal_replay -> 0x559523418b1es
u64s 11 type btree_ptr_v2 SPOS_MAX len 0 ver 0: seq a20e7ee3a59548b6 written 0 min_key R 346738:7053484335289139073:0 durability: 1 ptr: 1:2166957:512 gen 3
sectors_written == 0
transaction updates for bch2_journal_replay journal seq 4294967295
update: btree=dirents cached=0 0x559523418b1eS
old u64s 11 type btree_ptr_v2 SPOS_MAX len 0 ver 0: seq a20e7ee3a59548b6 written 0 min_key R 346738:7053484335289139073:0 durability: 1 ptr: 1:2166957:512 gen 3
new u64s 11 type btree_ptr_v2 SPOS_MAX len 0 ver 0: seq a20e7ee3a59548b6 written 0 min_key R 346738:7053484335289139073:0 durability: 1 ptr: 1:2166957:512 gen 3
bch2_btree_update_start: 237 callbacks suppressed
bch2_btree_update_start(): error Input/output error
bch2_btree_update_start(): error Input/output error
bch2_btree_update_start(): error Input/output error
bch2_journal_replay(): error while replaying key at btree inodes level 0: Input/output error
bch2_journal_replay(): error Input/output error
or mybe it still running, I see on htop 0.5% every now and then.. no errors on dmesg at all ...it was a prompt but it didn't write it, I pressed enter and it asked Y/N it looks like it get to the end and say unshutdown complete but not yet released the console and no activity in htop shrinkers thread is running once every 5 seconds for up to 1.5% and that's it memory use is dropping by around 1 MB every 2 hours or so, I guess it's doing something
I have an Arch Linux system that is installed to bcachefs. It is booting using EFISTUB with the kernel on a small EFI partition, everything else is bcachefs. It is a "home media server" running some Docker containers, Kodi, a few other things...
Just to give some relevant info about things I have done in the configuration:
It has two SSDs (/dev/sda1, /dev/sdd1) for foreground+promote and two HDDs (/dev/sdb, /dev/sdc) for background. Replicas=2 at the FS level. Replicas=1 set selectively on some directories. xxhash checksums, no compression.
At one point, I decided that I don't want the rootfs to go on the background devices anymore, so I did
bcachefs setattr --background_target=ssd /
(set the background to ssd on the root of the filesystem), and then set it back on specific directories likebcachefs setattr --background_target=hdd /srv
. I am not sure if this is a good idea, but oh well, it seemed to work.Kernel: 6.7.6-arch1-2
The issue is that I can no longer mount my filesystem, because it hangs during mount during journal replay. I am now trying to mount it from a live ISO to at least collect dmesg output.
Relevant
dmesg
output:bcachefs show-super
output:Running
bcachefs fsck -v /dev/sda1:/dev/sdb:/dev/sdc:/dev/sdd1
:Please let me know what I could do, if anything, to try to safely recover my filesystem?
Or if you'd like anything else from me to help you diagnose and fix this bug?