search

EOSIO / eos

An open source smart contract platform
https://developers.eos.io/manuals/eos
MIT License
11.28k stars 3.6k forks source link

Unlinkable block #5363

Closed williamleecn closed 5 years ago

williamleecn commented 6 years ago 
2018-08-22T04:25:32.647 thread-0   producer_plugin.cpp:309       on_incoming_block    ] 3030001 unlinkable_block_exception: Unlinkable block
unlinkable block
    {"id":"00bd1da28feb56b0d31c56383e230767315d10ec5f4dc0a9a328395596e8438b","previous":"00bd1da1591315791655cbf100df9ffd89064172219efb654e93b20875dc9128"}
    thread-0  fork_database.cpp:151 add
rethrow
    {}
    thread-0  controller.cpp:1036 push_block

running more than 10 days, but yesterday occur this error and sync stop

vjoke commented 6 years ago

which version? from the commit info, versions with 1.1.4 had resolved this issue.

williamleecn commented 6 years ago

newest ,v1.2.1

wanderingbort commented 6 years ago

@williamleecn Unlinkable block errors are a symptom of a problem, not the root problem. In the log that had unlinkable block errors there should be a single error that precedes the unlinkable block errors. That error will be the one we can use to identify the real issue at hand. Do you still have the logs before this error?

Irvingyao commented 6 years ago

i met the same error, it said '3030001 unlinkable_block_exception: Unlinkable block...'.

and i get another error ahead of it, see below:

''' 2018-08-23T12:21:28.811 thread-0 controller.cpp:1012 applyblock ] e.to detail_string(): 3030000 block_validate_exception: Block exception receipt does not match {"producer_receipt":{"status":"soft_fail","cpu_usage_us":4248,"net_usage_words":0,"trx":[0,"6929422a066f4c59a0427c330e66eb8d31890778f706f045651c8cc386a3a2b8"]},"validator_receipt":{"status":"executed","cpu_usage_us":4248,"net_usage_words":6,"trx":[0,"6929422a066f4c59a0427c330e66eb8d31890778f706f045651c8cc386a3a2b8"]}} thread-0 controller.cpp:989 apply_block 2018-08-23T12:21:28.832 thread-0 controller.cpp:1036 push_block ] 3030000 block_validate_exception: Block exception receipt does not match {"producer_receipt":{"status":"soft_fail","cpu_usage_us":4248,"net_usage_words":0,"trx":[0,"6929422a066f4c59a0427c330e66eb8d31890778f706f045651c8cc386a3a2b8"]},"validator_receipt":{"status":"executed","cpu_usage_us":4248,"net_usage_words":6,"trx":[0,"6929422a066f4c59a0427c330e66eb8d31890778f706f045651c8cc386a3a2b8"]}} thread-0 controller.cpp:989 apply_block

{}
thread-0  controller.cpp:1016 apply_block

2018-08-23T12:21:28.832 thread-0 producer_plugin.cpp:309 on_incoming_block ] 3030000 block_validate_exception: Block exception receipt does not match {"producer_receipt":{"status":"soft_fail","cpu_usage_us":4248,"net_usage_words":0,"trx":[0,"6929422a066f4c59a0427c330e66eb8d31890778f706f045651c8cc386a3a2b8"]},"validator_receipt":{"status":"executed","cpu_usage_us":4248,"net_usage_words":6,"trx":[0,"6929422a066f4c59a0427c330e66eb8d31890778f706f045651c8cc386a3a2b8"]}} thread-0 controller.cpp:989 apply_block

{}
thread-0  controller.cpp:1016 apply_block

rethrow {} thread-0 controller.cpp:1036 push_block '''

seems i have a bad block? is it the root cause of it??

Irvingyao commented 6 years ago

refer to another issue #4884 ,this might due to i mixed used bnet_plugin and net_plugin. it might lead some blocks crashed. i know bnet_plugin is a newer one. and it can only connect with those who also activate it. my question is bnet_plugin and net_plugin, which one is more recommended? Could i activate both of them at the same time? thanks

heifner commented 6 years ago
cc32d9 commented 6 years ago

I had a crash last night with a fresh v1.2.2. Config.ini:

p2p-peer-address = seed2.greymass.com:9876
p2p-peer-address = node869-mainnet.eosauthority.com:9393
p2p-peer-address = br.eosrio.io:9876
p2p-peer-address = p2p.libertyblock.io:9800
p2p-peer-address = mainnet.libertyblock.io:9800
p2p-peer-address = peering.mainnet.eoscanada.com:9876
p2p-peer-address = p2p.eossweden.se:9876

plugin = eosio::bnet_plugin
bnet-endpoint = 0.0.0.0:4321
bnet-threads = 4
bnet-connect = nodebnet-mainnet.eosauthority.com:19393
bnet-connect = p2p.eosio.cr:20747
bnet-connect = node.eosflare.io:5222

nodeos console log attached. net reports a bad block at 2018-08-23T23:10:45.080 received from eosauthority, and then p2p plugin reports all connections being rejected (which should not be the case, the network did not have interruptions). Then nodeos crashed 

*** Error in `/usr/local/eosio/bin/nodeos': double free or corruption (fasttop): 0x00007f7d140e4d90 ***

nodeos_1.2.2.crash.log

cc32d9 commented 6 years ago

Also once I commented out p2p peers and left only bnet peers, nodeos complained constantly about unlinkable blocks, and was not advancing anywhere. Then I commented out all bnet statements and activated only p2p statements, and nodeos continued processing the blockchain as expected.

Looks like something is fundamentally broken in bnet plugin.

heifner commented 6 years ago

@cc32d9 How easily can you reproduce this? Would you be willing to test a patch?

cc32d9 commented 6 years ago

@heifner quite easily over the weekend. The server runs already two nodeos instances in LXC containers over ZFS clones, so making a new test instance is a no-brainer.

heifner commented 6 years ago

@cc32d9 Could you try this: https://github.com/EOSIO/eos/pull/5417 which is branch: bnet-fix

cc32d9 commented 6 years ago

I deleted my previous message because I forgot to install compiled binaries, but in fact it was true: the new patch "server_version_string": "v1.2.0-52-gf2b49e8" does not solve the problem with unlinkable blocks. "last_irreversible_block_num": 13038555, and it's not moving.

2018-08-26T15:22:08.109 thread-0   controller.cpp:1036           push_block           ] 3030001 unlinkable_block_exception: Unlinkable block
unlinkable block
    {"id":"00c8d025a14382b21b8275a339376262d0269d7ef0cb8e2a43406cb6185851c5","previous":"00c8d024107e97e5fcdc795d3713a8ea58dbd0c47db1979c5722e22779787d84"}
    thread-0  fork_database.cpp:151 add
2018-08-26T15:22:08.110 thread-0   producer_plugin.cpp:309       on_incoming_block    ] 3030001 unlinkable_block_exception: Unlinkable block
unlinkable block
    {"id":"00c8d025a14382b21b8275a339376262d0269d7ef0cb8e2a43406cb6185851c5","previous":"00c8d024107e97e5fcdc795d3713a8ea58dbd0c47db1979c5722e22779787d84"}
    thread-0  fork_database.cpp:151 add
rethrow
    {}
    thread-0  controller.cpp:1036 push_block
2018-08-26T15:22:08.526 thread-0   controller.cpp:1036           push_block           ] 3030001 unlinkable_block_exception: Unlinkable block
unlinkable block
    {"id":"00c8d02638b637fe0a0b65e8bbce9cc906cce54a45c23b0c75b877433c17284d","previous":"00c8d025a14382b21b8275a339376262d0269d7ef0cb8e2a43406cb6185851c5"}
    thread-0  fork_database.cpp:151 add
2018-08-26T15:22:08.526 thread-0   producer_plugin.cpp:309       on_incoming_block    ] 3030001 unlinkable_block_exception: Unlinkable block
unlinkable block
    {"id":"00c8d02638b637fe0a0b65e8bbce9cc906cce54a45c23b0c75b877433c17284d","previous":"00c8d025a14382b21b8275a339376262d0269d7ef0cb8e2a43406cb6185851c5"}
    thread-0  fork_database.cpp:151 add
rethrow
    {}
    thread-0  controller.cpp:1036 push_block
2018-08-26T15:22:08.599 thread-0   controller.cpp:1036           push_block           ] 3030001 unlinkable_block_exception: Unlinkable block
unlinkable block
    {"id":"00c8d02638b637fe0a0b65e8bbce9cc906cce54a45c23b0c75b877433c17284d","previous":"00c8d025a14382b21b8275a339376262d0269d7ef0cb8e2a43406cb6185851c5"}
    thread-0  fork_database.cpp:151 add
2018-08-26T15:22:08.599 thread-0   producer_plugin.cpp:309       on_incoming_block    ] 3030001 unlinkable_block_exception: Unlinkable block
unlinkable block
    {"id":"00c8d02638b637fe0a0b65e8bbce9cc906cce54a45c23b0c75b877433c17284d","previous":"00c8d025a14382b21b8275a339376262d0269d7ef0cb8e2a43406cb6185851c5"}
    thread-0  fork_database.cpp:151 add
rethrow
    {}
    thread-0  controller.cpp:1036 push_block
2018-08-26T15:22:08.607 thread-0   controller.cpp:1036           push_block           ] 3030001 unlinkable_block_exception: Unlinkable block
unlinkable block
    {"id":"00c8d02638b637fe0a0b65e8bbce9cc906cce54a45c23b0c75b877433c17284d","previous":"00c8d025a14382b21b8275a339376262d0269d7ef0cb8e2a43406cb6185851c5"}
    thread-0  fork_database.cpp:151 add
2018-08-26T15:22:08.607 thread-0   producer_plugin.cpp:309       on_incoming_block    ] 3030001 unlinkable_block_exception: Unlinkable block
unlinkable block
    {"id":"00c8d02638b637fe0a0b65e8bbce9cc906cce54a45c23b0c75b877433c17284d","previous":"00c8d025a14382b21b8275a339376262d0269d7ef0cb8e2a43406cb6185851c5"}
    thread-0  fork_database.cpp:151 add
rethrow
    {}
    thread-0  controller.cpp:1036 push_block
2018-08-26T15:22:09.027 thread-0   controller.cpp:1036           push_block           ] 3030001 unlinkable_block_exception: Unlinkable block
unlinkable block
    {"id":"00c8d027e1b78a6317b31cc05beb292d465ca911855a86bcbcafd75a478e9e12","previous":"00c8d02638b637fe0a0b65e8bbce9cc906cce54a45c23b0c75b877433c17284d"}
    thread-0  fork_database.cpp:151 add
2018-08-26T15:22:09.027 thread-0   producer_plugin.cpp:309       on_incoming_block    ] 3030001 unlinkable_block_exception: Unlinkable block
unlinkable block
    {"id":"00c8d027e1b78a6317b31cc05beb292d465ca911855a86bcbcafd75a478e9e12","previous":"00c8d02638b637fe0a0b65e8bbce9cc906cce54a45c23b0c75b877433c17284d"}
    thread-0  fork_database.cpp:151 add
rethrow
    {}
    thread-0  controller.cpp:1036 push_block
2018-08-26T15:22:09.101 thread-0   controller.cpp:1036           push_block           ] 3030001 unlinkable_block_exception: Unlinkable block
unlinkable block
    {"id":"00c8d027e1b78a6317b31cc05beb292d465ca911855a86bcbcafd75a478e9e12","previous":"00c8d02638b637fe0a0b65e8bbce9cc906cce54a45c23b0c75b877433c17284d"}
    thread-0  fork_database.cpp:151 add
2018-08-26T15:22:09.101 thread-0   producer_plugin.cpp:309       on_incoming_block    ] 3030001 unlinkable_block_exception: Unlinkable block
unlinkable block
    {"id":"00c8d027e1b78a6317b31cc05beb292d465ca911855a86bcbcafd75a478e9e12","previous":"00c8d02638b637fe0a0b65e8bbce9cc906cce54a45c23b0c75b877433c17284d"}
    thread-0  fork_database.cpp:151 add
rethrow
    {}
    thread-0  controller.cpp:1036 push_block
2018-08-26T15:22:09.112 thread-0   controller.cpp:1036           push_block           ] 3030001 unlinkable_block_exception: Unlinkable block
unlinkable block
    {"id":"00c8d027e1b78a6317b31cc05beb292d465ca911855a86bcbcafd75a478e9e12","previous":"00c8d02638b637fe0a0b65e8bbce9cc906cce54a45c23b0c75b877433c17284d"}
    thread-0  fork_database.cpp:151 add
2018-08-26T15:22:09.112 thread-0   producer_plugin.cpp:309       on_incoming_block    ] 3030001 unlinkable_block_exception: Unlinkable block
unlinkable block
    {"id":"00c8d027e1b78a6317b31cc05beb292d465ca911855a86bcbcafd75a478e9e12","previous":"00c8d02638b637fe0a0b65e8bbce9cc906cce54a45c23b0c75b877433c17284d"}
    thread-0  fork_database.cpp:151 add
rethrow
    {}
    thread-0  controller.cpp:1036 push_block
cc32d9 commented 6 years ago

with one p2p peer enabled, it still keeps complaining about unlinkable blocks, but the head of the chain is moving forward.

cc32d9 commented 6 years ago

at least it doesn't segfault after re-syncing with mainnet

cc32d9 commented 6 years ago

when re-syncing finished, it can start with bnet peers only without complaints.

heifner commented 6 years ago

@cc32d9 Let me see if I follow correctly. When you are in state of unlinkable blocks, installing new code does not get you out of that state. However, using new code, if you start over all is well?

cc32d9 commented 6 years ago

@heifner , When you are in state of unlinkable blocks, installing new code does not get you out of that state -- correct.

using new code, if you start over all is well -- not exactly. From the same unlinkable state, I enabled a p2p peer, and it re-synched, and then it could run without the p2p peer,

heifner commented 6 years ago

@cc32d9 "p2p peer" == bnet_plugin or net_plugin?

cc32d9 commented 6 years ago

net_plugin. I added a legacy p2p peer, and the node got in-sync after a while.

heifner commented 6 years ago

@cc32d9 How long did you let bnet_plugin attempt to get you out of unlinkable blocks? How many threads are you running on bnet_plugin? How many bnet-connect entries?

cc32d9 commented 6 years ago

@heifner a couple of minutes. i can re-do it for longer period if needed. My config is above: 4 threads and 3 bnet peers.

heifner commented 6 years ago

@cc32d9 Thanks. Forgot the config was above, oops. If you could test again and give it say 30 minutes, I would appreciate it.

cc32d9 commented 6 years ago

it's running for about 30 minutes already, with the same errors. It keeps running, and I will check tomorrow again.

cc32d9 commented 6 years ago

it's been running for 6 hours, and didn't go any further.

heifner commented 6 years ago

@cc32d9 Can you share your log output?

datnv-ibl commented 6 years ago

I got this issue last night: unlinkable_block_exception, node stop and got dirty database when start again. Full log from startup to crash: https://drive.google.com/file/d/1pW6BiT8CUTfx14UqzAoc3f033D-cLguy/view?usp=sharing

heifner commented 6 years ago

@datnv-ibl What version of the eos.io?

datnv-ibl commented 6 years ago

It's v1.1.3

helperShang commented 6 years ago

It's v1.2.5

taokayan commented 5 years ago

Please try using latest version with p2p plugin. Please ask in eosio.stackexchange.com if you have further questions.

rameshb444 commented 5 years ago

Is this issue resolved?

I am using v1.4.1, I am getting the same error.

Logs :

warn 2018-11-20T06:22:30.728 thread-0 controller.cpp:1251 push_block ] 3030001 unlinkable_block_exception: Unlinkable block unlinkable block

{"id":"0034188318114e84a5682913e8776b6b224af635885ee3ccc275762b25911965","previous":"00341882315cfb6f565aaed395e0e3259d4cd61a51158d868f27f41c42fcce11"}
thread-0  fork_database.cpp:151 add

error 2018-11-20T06:22:30.728 thread-0 producer_plugin.cpp:316 on_incoming_block ] 3030001 unlinkable_block_exception: Unlinkable block unlinkable block {"id":"0034188318114e84a5682913e8776b6b224af635885ee3ccc275762b25911965","previous":"00341882315cfb6f565aaed395e0e3259d4cd61a51158d868f27f41c42fcce11"} thread-0 fork_database.cpp:151 add rethrow {} thread-0 controller.cpp:1251 push_block

jgiszczak commented 5 years ago

@rameshb444 The underlying issue is not resolved. There is a workaround. bnet_plugin is still unreliable. Do not use bnet-connect = in your config.ini. Use only p2p-peer-address = to connect peers.