heifner
commented
5 months ago Very interesting test failure. The nodes understanding of LIB (213) is ahead of its current understanding of HEAD (148). As it is applying blocks it has switched to a different fork that does not have a block that will become LIB. Although it is likely that the node will switch back the node doesn't know that yet.
Currently log_irreversible calls forkdb.fetch_branch( irreversible_block_id, new_lib_num) and moves LIB if the block has been validated. Instead, we need to fetch the branch from irreversible_block_id to HEAD since blocks could be validated but not be on this nodes understanding of best HEAD.
debug 2024-04-17T20:51:06.408 net-2 controller.cpp:4417 set_if_irreversible_ ] set irreversible block 148: 0000009406751a5c8e09cae1bccfd5363582f7e3a316531f8bacca6a31cbfa44, old 147: 00000093ab07f33365dea233f310bca8dba0917259ac60e88a9e7b2d043fc742
...
debug 2024-04-17T20:51:06.726 net-2 controller.cpp:4417 set_if_irreversible_ ] set irreversible block 213: 000000d5a766d19c52abb83c1dc26b9bc22b959b4b20a41bb9e6d3a4b3a9007e, old 212: 000000d4cd805f8daf2a272671f9e381b0ca99196a6897687171b5d35cb56609
debug 2024-04-17T20:51:06.726 net-2 net_plugin.cpp:3830 operator() ] validated block header, broadcasting immediately, connection - 4, blk num = 216, id = 000000d87db9c060fc5908230de92538cd5f979ac92930a0822588c16b07db09
debug 2024-04-17T20:51:06.726 net-2 net_plugin.cpp:3830 operator() ] validated block header, broadcasting immediately, connection - 5, blk num = 161, id = 000000a18b375d5bfff15d77276977384626398b0849afc77b5332f7210fa484
...
info 2024-04-17T20:51:06.841 nodeos controller.cpp:3379 log_applied ] Received block 0bd5dfb6da178186... #148 @ 2024-04-17T20:50:42.000 signed by defproducerd [trxs: 1, lib: 145, net: 144, cpu: 252, elapsed: 447035, time: 894421, latency: 24841 ms]
info 2024-04-17T20:51:06.841 nodeos controller.cpp:4145 operator() ] successfully switched fork to new head 000000940bd5dfb6da1781860b2c689f4fef99f5593661b0877d691d69613c37, removed {148:0000009406751a5c8e09cae1bccfd5363582f7e3a316531f8bacca6a31cbfa44}, applied {148:000000940bd5dfb6da1781860b2c689f4fef99f5593661b0877d691d69613c37}
debug 2024-04-17T20:51:06.841 nodeos net_plugin.cpp:4039 on_irreversible_bloc ] on_irreversible_block, blk num = 146, id = 00000092b716dbfb03a7b814a24f8af973a77f85640e099c3a751885ad3fba6d
debug 2024-04-17T20:51:06.841 nodeos net_plugin.cpp:3263 update_chain_info ] updating chain info lib 146, fork 148
debug 2024-04-17T20:51:06.841 trace-mx trace_api_plugin.cpp:155 operator() ] Waking up to handle lib: 146
debug 2024-04-17T20:51:06.841 nodeos net_plugin.cpp:4039 on_irreversible_bloc ] on_irreversible_block, blk num = 147, id = 00000093ab07f33365dea233f310bca8dba0917259ac60e88a9e7b2d043fc742
debug 2024-04-17T20:51:06.841 nodeos net_plugin.cpp:3263 update_chain_info ] updating chain info lib 147, fork 148
debug 2024-04-17T20:51:06.841 trace-mx trace_api_plugin.cpp:155 operator() ] Waking up to handle lib: 147
debug 2024-04-17T20:51:06.841 nodeos net_plugin.cpp:4039 on_irreversible_bloc ] on_irreversible_block, blk num = 148, id = 0000009406751a5c8e09cae1bccfd5363582f7e3a316531f8bacca6a31cbfa44
debug 2024-04-17T20:51:06.841 nodeos net_plugin.cpp:3263 update_chain_info ] updating chain info lib 148, fork 148
debug 2024-04-17T20:51:06.841 trace-mx trace_api_plugin.cpp:155 operator() ] Waking up to handle lib: 148
warn 2024-04-17T20:51:06.841 nodeos controller.cpp:3954 push_block ] 3020000 fork_database_exception: Fork database exception
removing the block and its descendants would remove the current head block
{}
nodeos fork_database.cpp:597 remove_impl
error 2024-04-17T20:51:06.842 nodeos producer_plugin.cpp:737 on_incoming_block ] Cannot recover from 3020000 fork_database_exception: Fork database exception
removing the block and its descendants would remove the current head block
{}
nodeos fork_database.cpp:597 remove_impl
rethrow
{}
nodeos controller.cpp:3954 push_block
. Shutting down.
https://github.com/AntelopeIO/spring/actions/runs/8728356303/job/23948345171