search

bitshares / bitshares-core

BitShares Blockchain node and command-line wallet
https://bitshares.github.io/
Other
1.17k stars 647 forks source link

Syncing issue around block 60444406 #2497

Open abitmore opened 3 years ago

abitmore commented 3 years ago

Bug Description

There was a syncing issue around block 60444406.

The witness who generated this block was out of sync:

2021-07-24T22:24:06 th_a:invoke handle_block         handle_block ] Got block: #60444399 039a4eef77b0463c94c13b135c442a5d539b83b5 time: 2021-07-24T22:24:06 transaction(s): 4 latency: 1
90 ms from: abc123  irreversible: 60444383 (-16)                        application.cpp:598
2021-07-24T22:24:07 th_a:invoke handle_transaction   handle_transaction ] Got 2 transactions from network                       application.cpp:660
2021-07-24T22:24:08 th_a:invoke handle_transaction   handle_transaction ] Got 3 transactions from network                       application.cpp:660
2021-07-24T22:24:12 th_a:Witness Block Production block_production_loo ] Generated block #60444400 with 6 transaction(s) and timestamp 2021-07-24T22:24:12 at time 2021-07-24T22:24:12
                witness.cpp:272
2021-07-24T22:24:31 th_a:invoke handle_block          _push_block ] Switching to fork: 039a4ef1bfec83cd809b767eee0a36087cfab8ee                 db_block.cpp:153
2021-07-24T22:24:31 th_a:invoke handle_block          _push_block ] popping block #60444400 039a4ef0944c7dc718c5a767b94d24e47718be4f                    db_block.cpp:159
2021-07-24T22:24:31 th_a:invoke handle_block          _push_block ] pushing block from fork #60444400 039a4ef0fd6db95988e7c1dc0fc63e30752094a9                  db_block.cpp:166
2021-07-24T22:24:31 th_a:invoke handle_block          _push_block ] pushing block from fork #60444401 039a4ef1bfec83cd809b767eee0a36087cfab8ee                  db_block.cpp:166
2021-07-24T22:24:32 th_a:Witness Block Production block_production_loo ] Generated block #60444406 with 0 transaction(s) and timestamp 2021-07-24T22:24:30 at time 2021-07-24T22:24:32
                witness.cpp:272
2021-07-24T22:24:32 th_a:invoke handle_transaction   handle_transaction ] Got 2 transactions from network                       application.cpp:660
2021-07-24T22:24:33 th_a:invoke handle_block         handle_block ] Got block: #60444407 039a4ef70d36957be2cb88d13f11972843bb977d time: 2021-07-24T22:24:33 transaction(s): 7 latency: 4
98 ms from: delegate.freedom  irreversible: 60444392 (-15)                      application.cpp:598

Most of other nodes got this block as a "sync block" so it did not show in default.log:

2021-07-24T22:24:24 th_a:invoke handle_block         handle_block ] Got block: #60444404 039a4ef408560fc22579dd03962eff0783f5c362 time: 2021-07-24T22:24:24 transaction(s): 5 latency: 747 ms from: btspp-witness  irreversible: 60444389 (-15)                 application.cpp:598
2021-07-24T22:24:26 th_a:invoke handle_transaction   handle_transaction ] Got 4 transactions from network                       application.cpp:660
2021-07-24T22:24:27 th_a:invoke handle_transaction   handle_transaction ] Got 5 transactions from network                       application.cpp:660
2021-07-24T22:24:27 th_a:invoke handle_block         handle_block ] Got block: #60444405 039a4ef513a98e67ff51be747c4983e71e2be1df time: 2021-07-24T22:24:27 transaction(s): 6 latency: 670 ms from: magicwallet.witness  irreversible: 60444390 (-15)                   application.cpp:598
2021-07-24T22:24:29 th_a:invoke handle_transaction   handle_transaction ] Got 2 transactions from network                       application.cpp:660
2021-07-24T22:24:31 th_a:invoke handle_transaction   handle_transaction ] Got 2 transactions from network                       application.cpp:660
2021-07-24T22:24:33 th_a:invoke handle_block         handle_block ] Got block: #60444407 039a4ef70d36957be2cb88d13f11972843bb977d time: 2021-07-24T22:24:33 transaction(s): 7 latency: 189 ms from: delegate.freedom  irreversible: 60444392 (-15)                      application.cpp:598
2021-07-24T22:24:34 th_a:invoke handle_transaction   handle_transaction ] Got 1 transactions from network                       application.cpp:660
2021-07-24T22:24:36 th_a:invoke handle_block         handle_block ] Got block: #60444408 039a4ef8553c978cce656047379ec3749ee1dfd9 time: 2021-07-24T22:24:36 transaction(s): 3 latency: 413 ms from: gbac-ety001  irreversible: 60444393 (-15)                   application.cpp:598

A few nodes got stuck: image image

2021-07-24T22:24:27 th_a:invoke handle_block         handle_block ] Got block: #60444405 039a4ef513a98e67ff51be747c4983e71e2be1df time: 2021-07-24T22:24:27 transaction(s): 6 latency: 232 ms from: magicwallet.witness  irreversible: 6044
4390 (-15)                   application.cpp:546
2021-07-24T22:24:28 th_a:invoke handle_transaction   handle_transaction ] Got 3 transactions from network                       application.cpp:615
2021-07-24T22:24:29 th_a:invoke handle_transaction   handle_transaction ] Got 1 transactions from network                       application.cpp:615
2021-07-24T22:24:31 th_a:invoke handle_transaction   handle_transaction ] Got 2 transactions from network                       application.cpp:615
2021-07-24T22:25:27 th_a:Witness Block Production block_production_loo ] Generated block #60444406 with 7 transaction(s) and timestamp 2021-07-24T22:25:27 at time 2021-07-24T22:25:27                  witness.cpp:277

Getting stuck is the issue. Ideally those nodes should recover automatically by themselves. But they didn't.

Update: another instance image

Impacts Describe which portion(s) of BitShares Core may be impacted by this bug. Please tick at least one box.

CORE TEAM TASK LIST