mwc-node, branch switch, sync issue

bayk commented 3 years ago

Here are what I see at the logs. Node is not able to sync because it ban it's peers.

20210330 08:13:18.521 INFO grin_servers::grin::sync::header_sync - sync: ban a fraud peer: 3.6.231.127:3414, claimed height: 709274, total difficulty: 138195857202944
20210330 08:13:18.615 INFO grin_servers::common::adapters - Received 32 block headers from 3.132.79.177:3414, height 690634, hash = 00078b665387, tip_processed = 707015
20210330 08:13:18.695 INFO grin_servers::common::adapters - Received 32 block headers from 3.132.79.177:3414, height 690666, hash = 000445ca847f, tip_processed = 707015
20210330 08:13:18.733 INFO grin_servers::common::adapters - Received 32 block headers from 3.132.79.177:3414, height 690698, hash = 000549da643a, tip_processed = 707015
20210330 08:13:18.775 INFO grin_servers::common::adapters - Received 32 block headers from 3.132.79.177:3414, height 690730, hash = 00040a197c09, tip_processed = 707015
20210330 08:13:18.776 INFO grin_servers::common::adapters - Received 32 block headers from 3.132.79.177:3414, height 690762, hash = 0001c2c9b2cf, tip_processed = 707015
20210330 08:13:18.851 INFO grin_servers::common::adapters - Received 32 block headers from 3.132.79.177:3414, height 690794, hash = 000e6d9ba82c, tip_processed = 707015
20210330 08:13:18.858 INFO grin_servers::common::adapters - Received 32 block headers from 3.132.79.177:3414, height 690826, hash = 001018994c6f, tip_processed = 707015
20210330 08:13:18.859 INFO grin_servers::common::adapters - Received 32 block headers from 3.132.79.177:3414, height 690858, hash = 00092075a8f4, tip_processed = 707015
20210330 08:13:18.929 INFO grin_servers::common::adapters - Received 32 block headers from 3.132.79.177:3414, height 690890, hash = 0004979eff07, tip_processed = 707015
20210330 08:13:18.935 INFO grin_servers::common::adapters - Received 32 block headers from 3.132.79.177:3414, height 690922, hash = 000b15f436b7, tip_processed = 707015
20210330 08:13:18.936 INFO grin_servers::common::adapters - Received 32 block headers from 3.132.79.177:3414, height 690954, hash = 0004bca056d2, tip_processed = 707015
20210330 08:13:18.973 INFO grin_servers::common::adapters - Received 32 block headers from 3.132.79.177:3414, height 690986, hash = 0005957894b1, tip_processed = 707015
20210330 08:13:19.010 INFO grin_servers::common::adapters - Received 32 block headers from 3.132.79.177:3414, height 691018, hash = 000c7f7dde77, tip_processed = 707015
20210330 08:13:19.026 INFO grin_servers::common::adapters - Received 32 block headers from 3.132.79.177:3414, height 691050, hash = 000162abc249, tip_processed = 707015
20210330 08:13:19.027 INFO grin_servers::common::adapters - Received 32 block headers from 3.132.79.177:3414, height 691082, hash = 000bcb8ecea7, tip_processed = 707015
20210330 08:13:19.085 INFO grin_servers::common::adapters - Received 32 block headers from 3.132.79.177:3414, height 691114, hash = 0005302b8d9f, tip_processed = 707015
20210330 08:13:28.527 INFO grin_p2p::peers - Banning peer 3.132.79.177:3414, ban_reason FraudHeight
20210330 08:13:28.527 INFO grin_servers::grin::sync::header_sync - sync: ban a fraud peer: 3.132.79.177:3414, claimed height: 709275, total difficulty: 138195878013025
20210330 08:13:28.694 INFO grin_servers::common::adapters - Received 32 block headers from 3.35.124.165:3414, height 690634, hash = 00078b665387, tip_processed = 707015
20210330 08:13:28.845 INFO grin_servers::common::adapters - Received 32 block headers from 3.35.124.165:3414, height 690666, hash = 000445ca847f, tip_processed = 707015
20210330 08:13:28.846 INFO grin_servers::common::adapters - Received 32 block headers from 3.35.124.165:3414, height 690698, hash = 000549da643a, tip_processed = 707015
20210330 08:13:28.998 INFO grin_servers::common::adapters - Received 32 block headers from 3.35.124.165:3414, height 690730, hash = 00040a197c09, tip_processed = 707015
20210330 08:13:29.007 INFO grin_servers::common::adapters - Received 32 block headers from 3.35.124.165:3414, height 690762, hash = 0001c2c9b2cf, tip_processed = 707015
20210330 08:13:29.009 INFO grin_servers::common::adapters - Received 32 block headers from 3.35.124.165:3414, height 690794, hash = 000e6d9ba82c, tip_processed = 707015
20210330 08:13:29.010 INFO grin_servers::common::adapters - Received 32 block headers from 3.35.124.165:3414, height 690826, hash = 001018994c6f, tip_processed = 707015
20210330 08:13:29.206 INFO grin_servers::common::adapters - Received 32 block headers from 3.35.124.165:3414, height 690858, hash = 00092075a8f4, tip_processed = 707015
20210330 08:13:29.210 INFO grin_servers::common::adapters - Received 32 block headers from 3.35.124.165:3414, height 690890, hash = 0004979eff07, tip_processed = 707015
20210330 08:13:29.244 INFO grin_servers::common::adapters - Received 32 block headers from 3.35.124.165:3414, height 690922, hash = 000b15f436b7, tip_processed = 707015
20210330 08:13:29.247 INFO grin_servers::common::adapters - Received 32 block headers from 3.35.124.165:3414, height 690954, hash = 0004bca056d2, tip_processed = 707015
20210330 08:13:29.249 INFO grin_servers::common::adapters - Received 32 block headers from 3.35.124.165:3414, height 690986, hash = 0005957894b1, tip_processed = 707015
20210330 08:13:29.251 INFO grin_servers::common::adapters - Received 32 block headers from 3.35.124.165:3414, height 691018, hash = 000c7f7dde77, tip_processed = 707015
20210330 08:13:29.386 INFO grin_servers::common::adapters - Received 32 block headers from 3.35.124.165:3414, height 691050, hash = 000162abc249, tip_processed = 707015
20210330 08:13:29.388 INFO grin_servers::common::adapters - Received 32 block headers from 3.35.124.165:3414, height 691082, hash = 000bcb8ecea7, tip_processed = 707015
20210330 08:13:29.394 INFO grin_servers::common::adapters - Received 32 block headers from 3.35.124.165:3414, height 691114, hash = 0005302b8d9f, tip_processed = 707015
20210330 08:13:38.527 INFO grin_p2p::peers - Banning peer 3.35.124.165:3414, ban_reason FraudHeight

The node is requesting the same headers in the loop.

(modified with formatting).

suemto commented 3 years ago

We will ban a peer if it claims a higher total work but can't send us the real headers (stale in 120 seconds).

https://github.com/mwcproject/mwc-node/blob/4.3.0/servers/src/grin/sync/header_sync.rs#L137-L153

Before going to other investigations, a first trying could be changing this 120s to a bigger value, to see whether it will continue receiving headers from the peer before ban it.

suemto commented 3 years ago

I can not reproduce the case of the banning. But the case of repeating receiving same headers from different peers here seems a behavior by the design:

The node is on a long fork, with height 707,015.
The peers have the highest height with 710,245, so this node enters Sync state and start requesting header from peers.
One peer answers with the 512 headers, from a common header (690,634). But since this node is on 707,015 height, all these 512 headers will not help this node to switch the header chain, it just store all these received headers on a fork, and still use its own header chain with Tip height 707,015.
Then, things repeat forever.

It's reasonable that this node will not switch the header chain because its current chain still wins in its own view. The behavior is correct.

A possible "fix" for this exception case is, if seeing such kind of long fork, the node will request all the remaining headers on one shot, instead of still per-512 headers.

A "benefit" of this "problem" could be a "consensus" of forbidding any reorg longer than 512 blocks (i.e. about 8.5 hours).

bayk commented 3 years ago

@suemto , I am thinking that let's keep it as it is. It is really great that attacker will not be able to do reorg silently by mining privately and then release very long branch. Forbidding any reorg longer than 512 blocks make sense for me.

bayk commented 3 years ago

I will update discord channel with that

bayk commented 2 months ago

Didn't see for a while, closing the issue

mwcproject / mwc-node

mwc-node, branch switch, sync issue #37