search

AntelopeIO / spring

C++ implementation of the Antelope protocol with Savanna consensus
Other
5 stars 2 forks source link

Test failure: nodeos_read_terminate_at_block_if_lr_test #216

Closed heifner closed 3 months ago

heifner commented 3 months ago

https://github.com/AntelopeIO/spring/actions/runs/9271552554/job/25507821117?pr=211

Looks like there may be an issue with a terminate-at-block that corresponds to a Savanna transition block when running in irreversible mode:

I removed a lot of the logging:

info  2024-05-28T15:03:40.607 nodeos    controller.cpp:1363           transition_to_savann ] Transitioning to savanna, IF Genesis Block 74, IF Critical Block 74
info  2024-05-28T15:03:40.607 nodeos    controller.cpp:1384           operator()           ] Transition to instant finality happening after block 75, First IF Proper Block 76
debug 2024-05-28T15:03:40.607 nodeos    net_plugin.cpp:3296           update_chain_info    ] updating chain info lib 74, head 75
debug 2024-05-28T15:03:40.607 net-0     net_plugin.cpp:3937           operator()           ] accepted signed_block : #75 20c2bf700e3c1065...
debug 2024-05-28T15:03:41.052 net-0     net_plugin.cpp:2676           bcast_block          ] bcast block 76
debug 2024-05-28T15:03:41.053 net-0     net_plugin.cpp:3867           operator()           ] posting block 76 to app thread
debug 2024-05-28T15:03:41.054 nodeos    producer_plugin.cpp:762       on_incoming_block    ] received incoming block 76 0000004cfb65ab3083e6a320a7556c0b0c7115ba1a43e44eee2738bb6789d7bd
debug 2024-05-28T15:03:41.054 nodeos    net_plugin.cpp:3296           update_chain_info    ] updating chain info lib 74, head 76
debug 2024-05-28T15:03:41.055 net-2     net_plugin.cpp:4032           operator()           ] signaled accepted_block_header, blk num = 76, id = 0000004cfb65ab3083e6a320a7556c0b0c7115ba1a43e44eee2738bb6789d7bd
debug 2024-05-28T15:03:41.059 nodeos    net_plugin.cpp:3296           update_chain_info    ] updating chain info lib 74, head 76
debug 2024-05-28T15:03:41.059 net-1     net_plugin.cpp:3937           operator()           ] accepted signed_block : #76 fb65ab3083e6a320...
debug 2024-05-28T15:03:41.062 net-0     net_plugin.cpp:3774           handle_message       ] ["localhost:9876 - b4655cf" - 8 127.0.0.1:34802] received vote: block #76:fb65ab3083e6a320.., strong, key mWkI-aEc9pyc6DY0..
debug 2024-05-28T15:03:41.079 vote-0    net_plugin.cpp:4087           bcast_vote_message   ] bcast received vote: block #76 fb65ab3083e6a320.., strong, key qVbh4IjYZpRGo8U_..
debug 2024-05-28T15:03:41.976 net-2     controller.cpp:4488           set_if_irreversible_ ] set irreversible block 74: 0000004a7fea56d32108135e71e6be9feb0097cd41cd6bdf2ce67a1b22338c2e, old 0: 0000000000000000000000000000000000000000000000000000000000000000
debug 2024-05-28T15:03:41.976 net-2     net_plugin.cpp:3862           operator()           ] validated block header, broadcasting immediately, connection - 1, blk num = 77, id = 0000004d5f6d7ec6f083dc2c594f8bf143ec88a66133f0d9c00fde8a7ef6a539
debug 2024-05-28T15:03:41.976 net-2     net_plugin.cpp:2676           bcast_block          ] bcast block 77
debug 2024-05-28T15:03:41.987 nodeos    producer_plugin.cpp:762       on_incoming_block    ] received incoming block 77 0000004d5f6d7ec6f083dc2c594f8bf143ec88a66133f0d9c00fde8a7ef6a539
debug 2024-05-28T15:03:41.988 nodeos    net_plugin.cpp:3296           update_chain_info    ] updating chain info lib 74, head 77
debug 2024-05-28T15:03:42.443 net-3     net_plugin.cpp:2676           bcast_block          ] bcast block 78
debug 2024-05-28T15:03:42.444 nodeos    producer_plugin.cpp:762       on_incoming_block    ] received incoming block 78 0000004e81700def2adcca1c01bc08b3c6bd4ca65dd027f07b8966dcc5e09b8d
debug 2024-05-28T15:03:42.444 nodeos    net_plugin.cpp:3296           update_chain_info    ] updating chain info lib 74, head 78
debug 2024-05-28T15:03:42.444 nodeos    net_plugin.cpp:3296           update_chain_info    ] updating chain info lib 74, head 78
debug 2024-05-28T15:03:42.888 net-1     net_plugin.cpp:3157           process_next_block_m ] ["localhost:9776:blk - 753b173" - 1 127.0.0.1:9776] received block 79, id 01b093b76791a6c3..., latency: -111ms, head 78
debug 2024-05-28T15:03:42.894 net-1     net_plugin.cpp:2676           bcast_block          ] bcast block 79
debug 2024-05-28T15:03:42.899 nodeos    net_plugin.cpp:3905           process_signed_block ] received signed_block: #79 block age in secs = 0, connection - 1, header validated, lib #74
debug 2024-05-28T15:03:42.899 nodeos    producer_plugin.cpp:762       on_incoming_block    ] received incoming block 79 0000004f01b093b76791a6c3f739ad5b7b28e512273d62d7cbb0d23635552eca
debug 2024-05-28T15:03:42.899 nodeos    net_plugin.cpp:3296           update_chain_info    ] updating chain info lib 74, head 79
debug 2024-05-28T15:03:42.899 net-0     net_plugin.cpp:4032           operator()           ] signaled accepted_block_header, blk num = 79, id = 0000004f01b093b76791a6c3f739ad5b7b28e512273d62d7cbb0d23635552eca
debug 2024-05-28T15:03:42.899 net-0     net_plugin.cpp:2676           bcast_block          ] bcast block 79
info  2024-05-28T15:03:42.901 nodeos    controller.cpp:3482           apply_block          ] Block 75 reached configured maximum block 75; terminating
info  2024-05-28T15:03:42.901 nodeos    main.cpp:165                  operator()           ] appbase quit called
debug 2024-05-28T15:03:42.902 trace-mx  trace_api_plugin.cpp:155      operator()           ] Waking up to handle lib: 75
debug 2024-05-28T15:03:42.902 nodeos    net_plugin.cpp:4096           on_irreversible_bloc ] on_irreversible_block, blk num = 75, id = 0000004b20c2bf700e3c1065a29fb16137b6d90ff6d3f3a68d8ba7ab74b6924e
debug 2024-05-28T15:03:42.902 nodeos    net_plugin.cpp:3310           update_chain_info    ] updating chain info lib 75, head 79
info  2024-05-28T15:03:42.902 nodeos    controller.cpp:3482           apply_block          ] Block 76 reached configured maximum block 75; terminating
info  2024-05-28T15:03:42.902 nodeos    main.cpp:165                  operator()           ] appbase quit called
debug 2024-05-28T15:03:42.902 nodeos    net_plugin.cpp:4096           on_irreversible_bloc ] on_irreversible_block, blk num = 76, id = 0000004cfb65ab3083e6a320a7556c0b0c7115ba1a43e44eee2738bb6789d7bd
debug 2024-05-28T15:03:42.902 nodeos    net_plugin.cpp:3310           update_chain_info    ] updating chain info lib 76, head 79
warn  2024-05-28T15:03:42.902 nodeos    controller.cpp:4037           push_block           ] 3020000 fork_database_exception: Fork database exception
cannot advance root to a block that has not yet been validated
    {}
    nodeos  fork_database.cpp:307 advance_root_impl

error 2024-05-28T15:03:42.902 nodeos    producer_plugin.cpp:817       on_incoming_block    ] Cannot recover from 3020000 fork_database_exception: Fork database exception
cannot advance root to a block that has not yet been validated
    {}
    nodeos  fork_database.cpp:307 advance_root_impl
rethrow
    {}
    nodeos  controller.cpp:4037 push_block
. Shutting down.
info  2024-05-28T15:03:42.902 nodeos    main.cpp:165                  operator()           ] appbase quit called
debug 2024-05-28T15:03:42.902 nodeos    net_plugin.cpp:3296           update_chain_info    ] updating chain info lib 74, head 79
heifner commented 3 months ago

Appears to still be failing after #228 https://github.com/AntelopeIO/spring/actions/runs/9374432600/job/25810863895

heifner commented 3 months ago

Still apparently not working: https://github.com/AntelopeIO/spring/actions/runs/9403942699

spoonincode commented 3 months ago

https://github.com/AntelopeIO/spring/actions/runs/9411410139/job/25924927296

heifner commented 3 months ago

https://github.com/AntelopeIO/spring/actions/runs/9411410139/job/25924927296

Requires #255