btcd v0.24 stuck on block 830326: Failed to process block ..be24f56e: assertion failed: view missing input ..20f56e2f:1

[0xB10C]

My btcd v0.24 node connected to my https://fork.observer instance seems to be stuck on block 830326. It knows about the chain tip and considers it a valid fork, but fails to process block 000000000000000000022b7ad737a424e1d98f9e4af52ed59a699603ff0a96b0 (830327).

relevant logs:

2024-02-13 20:23:34.478 [INF] SYNC: Processed 1 block in the last 15m51.54s (2413 transactions, height 830326, 2024-02-13 20:24:09 +0000 UTC, ~98 MiB cache)
[..]
2024-02-13 20:25:56.414 [ERR] SYNC: Failed to process block 000000000000000000008238554fd52cffc567328e8d87a20ac26a611db9b523: block 0000000000000000000357088139cba6dc295875d7d3b2dd1fd764705727e451 is not in the main chain
[..]
2024-02-13 20:27:09.470 [INF] CHAN: REORGANIZE: Block 00000000000000000001a22c1c720af27eb193d3bc95371df04404eabe24f56e is causing a reorganize.
2024-02-13 20:27:09.471 [INF] CHAN: Flushing UTXO cache of 98 MiB with 13370 entries to disk. For large sizes, this can take up to several minutes...
2024-02-13 20:27:09.832 [ERR] SYNC: Failed to process block 00000000000000000001a22c1c720af27eb193d3bc95371df04404eabe24f56e: assertion failed: view missing input 4538124686f31fdef2930639c5957e72af5796a078127548b58b783420f56e2f:1

from fork-observer:

Based on assertion failed: view missing input 4538124686f31fdef2930639c5957e72af5796a078127548b58b783420f56e2f:1, it seems like https://mempool.space/tx/305b9a9c63df1206dbd409189f0a66e5b01f5f5cb2984b633d574fff33e086d2 can't be applied as https://mempool.space/tx/4538124686f31fdef2930639c5957e72af5796a078127548b58b783420f56e2f#flow=&vout=1 wasn't added to the UTXO set.

Restarting btcd doesn't seem to fix this problem. I'm not sure yet if this is a local UTXO set inconsistency or a more widespread problem. I haven't seen anyone else having this problem.

edit: asked on twitter if anyone else is seeing this https://twitter.com/0xB10C/status/1757875766635757689

[0xB10C]

I think that's the relevant code section here:

https://github.com/btcsuite/btcd/blob/13152b35e191385a874294a9dbc902e48b1d71b0/blockchain/utxocache.go#L392-L406

[0xB10C]

2024-02-13 20:25:56.414 [ERR] SYNC: Failed to process block 000000000000000000008238554fd52cffc567328e8d87a20ac26a611db9b523: block 0000000000000000000357088139cba6dc295875d7d3b2dd1fd764705727e451 is not in the main chain

The block 0000000000000000000357088139cba6dc295875d7d3b2dd1fd764705727e451 is a stale block at height 829613 (‎2024-02-09 06:10:47 (6 days ago today)). Maybe this block was applied and then not correctly reverted during a reorg? I don't have logs from then. I only know that the block is considered a valid-fork by the btcd node.

Here's the block in hex and JSON: https://gist.github.com/0xB10C/27558c819cbec59c1d40a02e0c090454

[0xB10C]

fwiw: forgot to mention it in the OP, but I'm running -prune=1536 and blocksonly=1.

[Crypt-iQ]

This might be related to the recent view changes @kcalvinalvin ?

[0xB10C]

fwiw: my testnet btcd v0.24.0 node got stuck with a similar error. Very likely related.

2024-03-06 14:55:40.530 [ERR] SYNC: Failed to process block 0000000072ff4fe3ea6d6f2446e052356aa4b475e12d6506cc24c358cc7535c3: block 0000000000191f79513996a482aa0004456de7d1755337cf8d1bbdc73d730107 is not in the main chain
2024-03-06 14:55:40.741 [INF] SYNC: Rejected block 00000000000007806d2574368b8688293a792a9f3d52263585dbf38673265c46 from XXX (outbound): output 029651a9e0781274ac36dfb2472eee2aba7b0a0569a234f4b8419092add5d692:6 referenced from transaction 4de0102010fb2bc3be69819b4e6318018cbd1d5fd45c2903226b58436b827a02:0 either does not exist or has already been spent
...
2024-03-06 15:10:04.928 [INF] SYNC: Syncing to block height 2580758 from peer XXX
2024-03-06 15:10:33.612 [INF] CHAN: REORGANIZE: Block 000000000000002bcb26fe057159efc9a0cd79c1c8d6f620b508681478bc7677 is causing a reorganize.
2024-03-06 15:10:33.612 [INF] CHAN: Flushing UTXO cache of 98 MiB with 10086 entries to disk. For large sizes, this can take up to several minutes...
2024-03-06 15:10:33.754 [ERR] SYNC: Failed to process block 000000000000002bcb26fe057159efc9a0cd79c1c8d6f620b508681478bc7677: assertion failed: view missing input 029651a9e0781274ac36dfb2472eee2aba7b0a0569a234f4b8419092add5d692:6

Will re-sync it.