accounting keys with version=0 fix doesn't persist

[g2p]

This issue seems more cosmetic, reporting it because the that fsck code was updated recently

When mounting (with or without -o fsck,fix_errors):

09:52:55.882747: bcachefs (99…): recovering from clean shutdown, journal seq 58017678
09:52:55.950641: bcachefs (99…): accounting_read...
09:52:55.950701: invalid bkey u64s 6 type accounting 0:8:6 len 0 ver 0: btree btree=(unknown) 0
09:52:55.950721:   accounting key with version=0: delete?, fixing
09:52:55.966640: invalid bkey u64s 6 type accounting 0:9:6 len 0 ver 0: btree btree=(unknown) 0
09:52:55.966676:   accounting key with version=0: delete?, fixing
09:52:55.990640: invalid bkey u64s 6 type accounting 0:15:6 len 0 ver 0: btree btree=(unknown) 0
09:52:55.990674:   accounting key with version=0: delete?, fixing
09:52:55.990699: invalid bkey u64s 6 type accounting 0:16:6 len 0 ver 0: btree btree=(unknown) 0
09:52:55.990714:   accounting key with version=0: delete?, fixing
09:52:56.098647:  done
09:52:56.478634: bcachefs (99…): alloc_read... done

Sometimes during runtime (edit: when I run bcachefs fs usage or look at /sys/kernel/debug/bcachefs/99*/btrees/accounting/keys):

16:07:12.493477: invalid bkey u64s 6 type accounting 0:15:6 len 0 ver 0: btree btree=(unknown) 0
16:07:12.493605:   accounting key with version=0: delete?, fixing
16:07:12.493624: invalid bkey u64s 6 type accounting 0:16:6 len 0 ver 0: btree btree=(unknown) 0
16:07:12.493641:   accounting key with version=0: delete?, fixing

And again at shutdown:

09:47:35.541132: bcachefs (99…): shutting down
09:47:35.541183: bcachefs (99…): going read-only
09:47:35.581130: bcachefs (99…): finished waiting for writes to stop
09:47:35.605131: bcachefs (99…): flushing journal and stopping allocators, journal seq 58017677
09:47:35.605164: bcachefs (99…): flushing journal and stopping allocators complete, journal seq 58017677
09:47:35.613130: bcachefs (99…): shutdown complete, journal seq 58017678
09:47:35.669127: invalid bkey u64s 6 type accounting 0:8:6 len 0 ver 0: btree btree=(unknown) 0
09:47:35.669164:   accounting key with version=0: delete?, fixing
09:47:35.693128: invalid bkey u64s 6 type accounting 0:9:6 len 0 ver 0: btree btree=(unknown) 0
09:47:35.693159:   accounting key with version=0: delete?, fixing
09:47:35.721129: invalid bkey u64s 6 type accounting 0:15:6 len 0 ver 0: btree btree=(unknown) 0
09:47:35.721157:   accounting key with version=0: delete?, fixing
09:47:35.721170: invalid bkey u64s 6 type accounting 0:16:6 len 0 ver 0: btree btree=(unknown) 0
09:47:35.721181:   accounting key with version=0: delete?, fixing
09:47:35.877129: bcachefs (99…): marking filesystem clean
09:47:35.897188: bcachefs (99…): shutdown complete

[koverstreet]

That report makes it look like we're still generating version=0 keys, and the patch making bkey errors AUTOFIX broke the emergency shutdown paths that would've given us more info.

Can you try the current bcachefs-for-upstream branch?

[g2p]

No change with the current bcachefs-for-upstream (0f25eb4b60771f08fbcca878a8f7f88086d0c885).

I get the messages, and the autofixing, on mount, umount, and when looking at filesystem usage. Still no idea where these are generated.

[g2p]

Is it expected to have accounting keys with btree=(unknown)? This is a common point between these zombie keys.

[g2p]

I've switched to 406358ec9c87f818dae2c04b54ba2448890a9567 "bcachefs: Avoid bch2_btree_id_str()" on bcachefs-testing, so I get slightly more detailed error messages:

2024-10-10T10:56:17.826733: bcachefs (99…): starting version 1.13: inode_has_child_snapshots opts=compression=lz4,background_compression=zstd,foreground_target=ssd,background_target=hdd,promote_target=ssd,gc_reserve_percent=5,verbose
2024-10-10T10:56:17.826824: bcachefs (99…): recovering from clean shutdown, journal seq 58265289
2024-10-10T10:56:17.914710: bcachefs (99…): accounting_read...
2024-10-10T10:56:17.914745: invalid bkey u64s 6 type accounting 0:8:6 len 0 ver 0: btree btree=(unknown btree 134217728) 0
2024-10-10T10:56:17.914765:   accounting key with version=0: delete?, fixing
2024-10-10T10:56:18.062711:  done
2024-10-10T10:56:18.394730: bcachefs (99…): alloc_read... done
2024-10-10T10:56:18.394808: bcachefs (99…): stripes_read... done
2024-10-10T10:56:18.394825: bcachefs (99…): snapshots_read... done
2024-10-10T10:56:18.474727: bcachefs (99…): journal_replay... done
2024-10-10T10:56:18.474769: bcachefs (99…): resume_logged_ops... done
2024-10-10T10:56:18.474796: bcachefs (99…): delete_dead_inodes... done
2024-10-10T10:56:18.474815: bcachefs (99…): going read-write
2024-10-10T10:56:18.474830: bcachefs (99…): done starting filesystem

134217728 is 2^27

Some of the other keys have been staying fixed, and this is the last one.

The messages from before the reboot, at umount time:

oct. 10 10:50:55: bcachefs (99…): shutting down
oct. 10 10:50:55: bcachefs (99…): going read-only
oct. 10 10:50:55: bcachefs (99…): finished waiting for writes to stop
oct. 10 10:50:55: bcachefs (99…): flushing journal and stopping allocators, journal seq 58265288
oct. 10 10:50:55: bcachefs (99…): flushing journal and stopping allocators complete, journal seq 58265288
oct. 10 10:50:55: bcachefs (99…): shutdown complete, journal seq 58265289
oct. 10 10:50:55: invalid bkey u64s 6 type accounting 0:8:6 len 0 ver 0: btree btree=(unknown) 0
oct. 10 10:50:55:   accounting key with version=0: delete?, fixing
oct. 10 10:50:55: bcachefs (99…): marking filesystem clean
oct. 10 10:50:55: bcachefs (99…): shutdown complete

[g2p]

Though if I look for "unknown btree" I find more:

grep 'unknown btree' /sys/kernel/debug/bcachefs/*/btrees/accounting/keys

/sys/kernel/debug/bcachefs/99…/btrees/accounting/keys:u64s 6 type accounting 0:1:6 len 0 ver 223039708970619963: btree btree=(unknown btree 16777216) 0
/sys/kernel/debug/bcachefs/99…/btrees/accounting/keys:u64s 6 type accounting 0:2:6 len 0 ver 211017940890302657: btree btree=(unknown btree 33554432) 0
/sys/kernel/debug/bcachefs/99…/btrees/accounting/keys:u64s 6 type accounting 0:3:6 len 0 ver 200599076079675518: btree btree=(unknown btree 50331648) 0
/sys/kernel/debug/bcachefs/99…/btrees/accounting/keys:u64s 6 type accounting 0:4:6 len 0 ver 223471636651746005: btree btree=(unknown btree 67108864) 0
/sys/kernel/debug/bcachefs/99…/btrees/accounting/keys:u64s 6 type accounting 0:5:6 len 0 ver 200599956547968954: btree btree=(unknown btree 83886080) 0
/sys/kernel/debug/bcachefs/99…/btrees/accounting/keys:u64s 6 type accounting 0:6:6 len 0 ver 200599956547969038: btree btree=(unknown btree 100663296) 0
/sys/kernel/debug/bcachefs/99…/btrees/accounting/keys:u64s 6 type accounting 0:7:6 len 0 ver 223036148442753235: btree btree=(unknown btree 117440512) 0
/sys/kernel/debug/bcachefs/99…/btrees/accounting/keys:u64s 6 type accounting 0:10:6 len 0 ver 223474973841295589: btree btree=(unknown btree 167772160) 0
/sys/kernel/debug/bcachefs/99…/btrees/accounting/keys:u64s 6 type accounting 0:11:6 len 0 ver 218858171760920748: btree btree=(unknown btree 184549376) 0
/sys/kernel/debug/bcachefs/99…/btrees/accounting/keys:u64s 6 type accounting 0:12:6 len 0 ver 210472926720304453: btree btree=(unknown btree 201326592) 0
/sys/kernel/debug/bcachefs/99…/btrees/accounting/keys:u64s 6 type accounting 0:13:6 len 0 ver 223471748320889715: btree btree=(unknown btree 218103808) 0
/sys/kernel/debug/bcachefs/99…/btrees/accounting/keys:u64s 6 type accounting 0:14:6 len 0 ver 223450127455510715: btree btree=(unknown btree 234881024) 0
/sys/kernel/debug/bcachefs/99…/btrees/accounting/keys:u64s 6 type accounting 0:18:6 len 0 ver 223471726846060559: btree btree=(unknown btree 301989888) 0
/sys/kernel/debug/bcachefs/99…/btrees/accounting/keys:u64s 6 type accounting 0:19:6 len 0 ver 200619215181342260: btree btree=(unknown btree 318767104) 0
/sys/kernel/debug/bcachefs/99…/btrees/accounting/keys:u64s 6 type accounting 0:20:6 len 0 ver 199355051687280790: btree btree=(unknown btree 335544320) 0
/sys/kernel/debug/bcachefs/f4…/btrees/accounting/keys:u64s 6 type accounting 0:1:6 len 0 ver 56195352100988: btree btree=(unknown btree 16777216) 0
/sys/kernel/debug/bcachefs/f4…/btrees/accounting/keys:u64s 6 type accounting 0:2:6 len 0 ver 21474836514: btree btree=(unknown btree 33554432) 0
/sys/kernel/debug/bcachefs/f4…/btrees/accounting/keys:u64s 6 type accounting 0:4:6 len 0 ver 4294969487: btree btree=(unknown btree 67108864) 0
/sys/kernel/debug/bcachefs/f4…/btrees/accounting/keys:u64s 6 type accounting 0:7:6 len 0 ver 197568496461: btree btree=(unknown btree 117440512) 0
/sys/kernel/debug/bcachefs/f4…/btrees/accounting/keys:u64s 6 type accounting 0:8:6 len 0 ver 17179874433: btree btree=(unknown btree 134217728) 0
/sys/kernel/debug/bcachefs/f4…/btrees/accounting/keys:u64s 6 type accounting 0:9:6 len 0 ver 17179874295: btree btree=(unknown btree 150994944) 0
/sys/kernel/debug/bcachefs/f4…/btrees/accounting/keys:u64s 6 type accounting 0:10:6 len 0 ver 73014444151: btree btree=(unknown btree 167772160) 0
/sys/kernel/debug/bcachefs/f4…/btrees/accounting/keys:u64s 6 type accounting 0:11:6 len 0 ver 17179874043: btree btree=(unknown btree 184549376) 0
/sys/kernel/debug/bcachefs/f4…/btrees/accounting/keys:u64s 6 type accounting 0:12:6 len 0 ver 141733921266: btree btree=(unknown btree 201326592) 0
/sys/kernel/debug/bcachefs/f4…/btrees/accounting/keys:u64s 6 type accounting 0:13:6 len 0 ver 25769803810: btree btree=(unknown btree 218103808) 0
/sys/kernel/debug/bcachefs/f4…/btrees/accounting/keys:u64s 6 type accounting 0:14:6 len 0 ver 575525617968: btree btree=(unknown btree 234881024) 0
/sys/kernel/debug/bcachefs/f4…/btrees/accounting/keys:u64s 6 type accounting 0:15:6 len 0 ver 17179874170: btree btree=(unknown btree 251658240) 0
/sys/kernel/debug/bcachefs/f4…/btrees/accounting/keys:u64s 6 type accounting 0:16:6 len 0 ver 51539608564: btree btree=(unknown btree 268435456) 0
/sys/kernel/debug/bcachefs/f4…/btrees/accounting/keys:u64s 6 type accounting 0:17:6 len 0 ver 64424509710: btree btree=(unknown btree 285212672) 0
/sys/kernel/debug/bcachefs/f4…/btrees/accounting/keys:u64s 6 type accounting 0:18:6 len 0 ver 73014444254: btree btree=(unknown btree 301989888) 0
/sys/kernel/debug/bcachefs/f4…/btrees/accounting/keys:u64s 6 type accounting 0:20:6 len 0 ver 25769803986: btree btree=(unknown btree 335544320) 0
/sys/kernel/debug/bcachefs/ff…/btrees/accounting/keys:u64s 6 type accounting 0:1:6 len 0 ver 33170848470794420: btree btree=(unknown btree 16777216) 0
/sys/kernel/debug/bcachefs/ff…/btrees/accounting/keys:u64s 6 type accounting 0:2:6 len 0 ver 33180920169113601: btree btree=(unknown btree 33554432) 0
/sys/kernel/debug/bcachefs/ff…/btrees/accounting/keys:u64s 6 type accounting 0:3:6 len 0 ver 11247896577966496: btree btree=(unknown btree 50331648) 0
/sys/kernel/debug/bcachefs/ff…/btrees/accounting/keys:u64s 6 type accounting 0:4:6 len 0 ver 33154128163111092: btree btree=(unknown btree 67108864) 0
/sys/kernel/debug/bcachefs/ff…/btrees/accounting/keys:u64s 6 type accounting 0:7:6 len 0 ver 30495384493105285: btree btree=(unknown btree 117440512) 0
/sys/kernel/debug/bcachefs/ff…/btrees/accounting/keys:u64s 6 type accounting 0:8:6 len 0 ver 103079215149: btree btree=(unknown btree 134217728) 0
/sys/kernel/debug/bcachefs/ff…/btrees/accounting/keys:u64s 6 type accounting 0:9:6 len 0 ver 98784261363: btree btree=(unknown btree 150994944) 0
/sys/kernel/debug/bcachefs/ff…/btrees/accounting/keys:u64s 6 type accounting 0:10:6 len 0 ver 33178115555461098: btree btree=(unknown btree 167772160) 0
/sys/kernel/debug/bcachefs/ff…/btrees/accounting/keys:u64s 6 type accounting 0:11:6 len 0 ver 33081513151041875: btree btree=(unknown btree 184549376) 0
/sys/kernel/debug/bcachefs/ff…/btrees/accounting/keys:u64s 6 type accounting 0:12:6 len 0 ver 32697890967152940: btree btree=(unknown btree 201326592) 0
/sys/kernel/debug/bcachefs/ff…/btrees/accounting/keys:u64s 6 type accounting 0:13:6 len 0 ver 33188698354876550: btree btree=(unknown btree 218103808) 0
/sys/kernel/debug/bcachefs/ff…/btrees/accounting/keys:u64s 6 type accounting 0:14:6 len 0 ver 33103967240075803: btree btree=(unknown btree 234881024) 0
/sys/kernel/debug/bcachefs/ff…/btrees/accounting/keys:u64s 6 type accounting 0:15:6 len 0 ver 98784261187: btree btree=(unknown btree 251658240) 0
/sys/kernel/debug/bcachefs/ff…/btrees/accounting/keys:u64s 6 type accounting 0:16:6 len 0 ver 1168231105356: btree btree=(unknown btree 268435456) 0
/sys/kernel/debug/bcachefs/ff…/btrees/accounting/keys:u64s 6 type accounting 0:17:6 len 0 ver 10158306324667245: btree btree=(unknown btree 285212672) 0
/sys/kernel/debug/bcachefs/ff…/btrees/accounting/keys:u64s 6 type accounting 0:18:6 len 0 ver 33130694821546073: btree btree=(unknown btree 301989888) 0
/sys/kernel/debug/bcachefs/ff…/btrees/accounting/keys:u64s 6 type accounting 0:20:6 len 0 ver 94489308003: btree btree=(unknown btree 335544320) 0

(The btree ids aren't all powers of two, but they all have at least 24 trailing zeroes)

The first time I grep there's just one that gets reported in dmesg

11:12:17.354721: invalid bkey u64s 6 type accounting 0:8:6 len 0 ver 0: btree btree=(unknown btree 134217728) 0
11:12:17.354988:   accounting key with version=0: delete?, fixing