search

pocketnetteam / pocketnet.core

Decentralized social network based on the blockchain
https://pocketnet.app
Apache License 2.0
115 stars 28 forks source link

Unrecoverable fork at 2140569 and forked checkpoint #571

Closed HiHat closed 8 months ago

HiHat commented 1 year ago

Describe the bug Unrecoverable fork issued at 2140569. Current checkpoint (from 2023-03-18) provides forked chain at height 2140978 (best block: a34b9c3b175e89f394971041aeab914ab759a8437de15cf0230efaac42036cd5).

To Reproduce

  1. Install forked checkpoint
  2. Run node and wait for outbound connections to other nodes
  3. Following messages in debug.log: 
    2023-03-18T21:35:33Z New outbound peer connected: version: 70016, blocks=2142704, peer=0, peeraddr=202.61.253.55:37070 (full-relay)
2023-03-18T21:35:33Z Synchronizing blockheaders, height: 2142704 (~100.00%)
2023-03-18T21:35:33Z     Block checked with result 1: Height: 2140569 BH: aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-18T21:35:33Z Rollback current block to prev at height 2140977
2023-03-18T21:35:37Z Rollback current block to prev at height 2140976
...
2023-03-18T22:02:25Z Rollback current block to prev at height 2140569
2023-03-18T22:02:29Z Rollback current block to prev at height 2140568
2023-03-18T22:02:36Z Warning: SocialConsensus type:305 validate tx:0597aa5dc60e70a16d497b89d58ef337749f4cd751ba967e232099d905c35bdc blk:aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed with result:23 at height:2140569
2023-03-18T22:02:36Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-18T22:02:36Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-18T22:02:47Z     Block validated: 2140569 BH: 7aec19f7c5f7673fe46c3ba56ec8d3ed0ab7bd3da4354d62b11c8a48231b97a9
2023-03-18T22:03:04Z     Block indexed: 2140569 BH: 7aec19f7c5f7673fe46c3ba56ec8d3ed0ab7bd3da4354d62b11c8a48231b97a9
2023-03-18T22:03:04Z +++ Block connected to chain: 2140569 BH: 7aec19f7c5f7673fe46c3ba56ec8d3ed0ab7bd3da4354d62b11c8a48231b97a9
2023-03-18T22:03:05Z     Block validated: 2140570 BH: 5fbd6e7ceda9118ad4ad366c58c52af0cbfbea838afe10b1d45b84a10f4e3200
2023-03-18T22:03:05Z     Block indexed: 2140570 BH: 5fbd6e7ceda9118ad4ad366c58c52af0cbfbea838afe10b1d45b84a10f4e3200
2023-03-18T22:03:05Z +++ Block connected to chain: 2140570 BH: 5fbd6e7ceda9118ad4ad366c58c52af0cbfbea838afe10b1d45b84a10f4e3200
...
2023-03-18T22:17:45Z     Block validated: 2140978 BH: a34b9c3b175e89f394971041aeab914ab759a8437de15cf0230efaac42036cd5
2023-03-18T22:17:46Z     Block indexed: 2140978 BH: a34b9c3b175e89f394971041aeab914ab759a8437de15cf0230efaac42036cd5
2023-03-18T22:17:46Z +++ Block connected to chain: 2140978 BH: a34b9c3b175e89f394971041aeab914ab759a8437de15cf0230efaac42036cd5
2023-03-18T22:17:46Z Disconnecting and discouraging peer 0!
2023-03-18T22:17:47Z New outbound peer connected: version: 70016, blocks=2142742, peer=2, peeraddr=65.21.56.203:37070 (full-relay)
2023-03-18T22:17:47Z Synchronizing blockheaders, height: 2142742 (~100.00%)
2023-03-18T22:17:47Z     Block checked with result 1: Height: 2140569 BH: aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-18T22:17:48Z Rollback current block to prev at height 2140977
2023-03-18T22:17:52Z Rollback current block to prev at height 2140976
...

Expected behavior Rollback to 2140568 block then validating and connecting 2140569 block to chain with valid hash (aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592).

Host and node version:

andyoknen commented 1 year ago

We are dealing with the problem

the-real-vortex-v commented 1 year ago

When you do a bug report like this in future. I highly suggest giving a list of the connected nodes. Old versions of nodes can influence and cause this issues (and have done in the past).

Where there any versions of the node software lower than V21 connected at the time?

@AOK can you give us an update of what's possibly causing this because lots of node operators are getting this or similar at the moment. An update in the node chat would be good.

HiHat commented 1 year ago

When you do a bug report like this in future. I highly suggest giving a list of the connected nodes.

According to provided report and log there were no connections as it was cold start from forked checkpoint (at height 2140978) obtained from snapshot.pocketnet.app! If we see at first line of log:

2023-03-18T21:35:33Z New outbound peer connected: version: 70016, blocks=2142704, peer=0, peeraddr=202.61.253.55:37070 (full-relay)

Then we found that first connection (due to peer=0) was made to 202.61.253.55 peer with strange 70016 version! Unfortunately node software versions (like 0.21.1) not dumped to logs! Currently 202.61.253.55 peer hosted 0.21.1 version.

HiHat commented 1 year ago

If we analyze log further then we found that fork was detected and node began rollback to 2140568 best block. Unfortunately 2140569 block was not correctly validated and new fork began:

2023-03-18T22:02:36Z Warning: SocialConsensus type:305 validate tx:0597aa5dc60e70a16d497b89d58ef337749f4cd751ba967e232099d905c35bdc blk:aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed with result:23 at height:2140569
2023-03-18T22:02:36Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-18T22:02:36Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-18T22:02:47Z     Block validated: 2140569 BH: 7aec19f7c5f7673fe46c3ba56ec8d3ed0ab7bd3da4354d62b11c8a48231b97a9
2023-03-18T22:03:04Z     Block indexed: 2140569 BH: 7aec19f7c5f7673fe46c3ba56ec8d3ed0ab7bd3da4354d62b11c8a48231b97a9
2023-03-18T22:03:04Z +++ Block connected to chain: 2140569 BH: 7aec19f7c5f7673fe46c3ba56ec8d3ed0ab7bd3da4354d62b11c8a48231b97a9
andyoknen commented 1 year ago

At the moment, together with @lostystyg, we are studying this problem, although at our stand the error is different - number 12, which indicates that there is no necessary data to connect the block from the main chain and (here's the problem) goes into fork again.

HiHat commented 1 year ago

I can share forked checkpoint (from 2023-03-18) to help reproduce error number 23!

andyoknen commented 1 year ago

We have one copy for analysis, but please leave yours just in case. After yesterday's study, we have so far come to the preliminary conclusion that the forks are related to the late update of some nodes, as some releases include consensus changes (adding new types of transactions or changing their behavior). Maybe you still have a log file that shows the version of the node starting from 0.20.29?

HiHat commented 1 year ago

I have 2 log files with node forking (banner.log and debug.log). Node forked at 2023-03-17 10:06 UTC.

In debug.log was following lines about forking (time in UTC):

2023-03-16T21:58:53Z New outbound peer connected: version: 70016, blocks=2139830, peer=0, peeraddr=95.31.45.162:37070 (block-relay)
2023-03-16T21:58:53Z Leaving InitialBlockDownload (latching to false)
2023-03-16T21:59:24Z New outbound peer connected: version: 70016, blocks=2139830, peer=1, peeraddr=91.206.15.176:37070 (block-relay)
2023-03-16T21:59:27Z New outbound peer connected: version: 70016, blocks=2139830, peer=2, peeraddr=73.96.245.93:37070 (full-relay)
2023-03-16T21:59:29Z New outbound peer connected: version: 70016, blocks=2139830, peer=3, peeraddr=81.169.156.24:37070 (full-relay)
2023-03-16T21:59:32Z New outbound peer connected: version: 70016, blocks=2139831, peer=4, peeraddr=67.140.183.89:37070 (full-relay)
2023-03-16T21:59:34Z New outbound peer connected: version: 70016, blocks=2139831, peer=5, peeraddr=135.181.196.243:37070 (full-relay)
2023-03-16T21:59:35Z New outbound peer connected: version: 70016, blocks=2139831, peer=6, peeraddr=94.190.60.151:37070 (full-relay)
2023-03-16T21:59:36Z New outbound peer connected: version: 70016, blocks=2139831, peer=7, peeraddr=162.246.52.155:37070 (full-relay)
2023-03-16T21:59:54Z New outbound peer connected: version: 70016, blocks=2139831, peer=9, peeraddr=85.236.190.129:37070 (full-relay)
2023-03-16T22:00:06Z New outbound peer connected: version: 70016, blocks=2139832, peer=10, peeraddr=68.118.120.152:37070 (full-relay)
2023-03-16T22:21:28Z New outbound peer connected: version: 70016, blocks=2139858, peer=23, peeraddr=81.169.156.24:37070 (full-relay)
...
2023-03-16T22:51:31Z New outbound peer connected: version: 70016, blocks=2139888, peer=45, peeraddr=109.195.147.45:37070 (full-relay)
2023-03-16T23:06:33Z New outbound peer connected: version: 70015, blocks=2139276, peer=57, peeraddr=77.173.8.2:37070 (full-relay)
2023-03-16T23:06:35Z New outbound peer connected: version: 70016, blocks=2139903, peer=58, peeraddr=37.48.87.107:37070 (full-relay)
...
2023-03-17T02:27:30Z New outbound peer connected: version: 70016, blocks=2140114, peer=197, peeraddr=188.18.52.142:37070 (full-relay)
...
2023-03-17T04:14:20Z New outbound peer connected: version: 70016, blocks=2140223, peer=262, peeraddr=65.21.252.135:37070 (full-relay)
...
2023-03-17T05:20:49Z Potential stale tip detected, will try using extra outbound peer (last tip update: 366 seconds ago)
2023-03-17T05:20:55Z New outbound peer connected: version: 70016, blocks=2140278, peer=310, peeraddr=142.116.40.18:37070 (full-relay)
2023-03-17T05:27:32Z New outbound peer connected: version: 70016, blocks=2140284, peer=316, peeraddr=140.99.153.138:37070 (full-relay)
...
2023-03-17T10:06:04Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:06:04Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:06:04Z Disconnecting and discouraging peer 316!
2023-03-17T10:06:05Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:06:05Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:06:05Z Disconnecting and discouraging peer 0!
2023-03-17T10:06:05Z New outbound peer connected: version: 70016, blocks=2133670, peer=480, peeraddr=209.55.111.169:37070 (full-relay)
2023-03-17T10:06:05Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:06:05Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:06:05Z Disconnecting and discouraging peer 1!
2023-03-17T10:06:08Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:06:08Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:06:08Z Disconnecting and discouraging peer 7!
2023-03-17T10:06:09Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:06:09Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:06:09Z Disconnecting and discouraging peer 32!
2023-03-17T10:06:09Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:06:09Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:06:09Z Disconnecting and discouraging peer 111!
2023-03-17T10:06:13Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:06:13Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:06:13Z Disconnecting and discouraging peer 315!
2023-03-17T10:06:14Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:06:14Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:06:14Z Disconnecting and discouraging peer 4!
2023-03-17T10:06:15Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:06:15Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:06:15Z Disconnecting and discouraging peer 6!
2023-03-17T10:06:15Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:06:15Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:06:15Z Disconnecting and discouraging peer 9!
2023-03-17T10:06:16Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:06:16Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:06:16Z Disconnecting and discouraging peer 58!
2023-03-17T10:06:17Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:06:17Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:06:17Z Disconnecting and discouraging peer 197!
2023-03-17T10:06:17Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:06:17Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:06:17Z Disconnecting and discouraging peer 262!
2023-03-17T10:06:19Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:06:19Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:06:19Z Disconnecting and discouraging peer 397!
2023-03-17T10:06:29Z New outbound peer connected: version: 70016, blocks=2140569, peer=484, peeraddr=135.181.196.243:37070 (full-relay)
2023-03-17T10:06:29Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:06:29Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:06:29Z Disconnecting and discouraging peer 484!
2023-03-17T10:06:30Z New outbound peer connected: version: 70016, blocks=2140568, peer=486, peeraddr=89.106.125.173:37070 (full-relay)
2023-03-17T10:07:00Z New outbound peer connected: version: 70016, blocks=2126978, peer=488, peeraddr=185.183.197.82:37070 (full-relay)
2023-03-17T10:07:07Z New outbound peer connected: version: 70016, blocks=2140568, peer=485, peeraddr=83.151.14.174:37070 (full-relay)
2023-03-17T10:07:12Z New outbound peer connected: version: 70016, blocks=2140570, peer=489, peeraddr=172.83.108.38:37070 (full-relay)
2023-03-17T10:07:14Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:07:14Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:07:14Z Disconnecting and discouraging peer 489!
2023-03-17T10:07:32Z New outbound peer connected: version: 70016, blocks=2140571, peer=491, peeraddr=109.197.196.106:37070 (full-relay)
2023-03-17T10:07:34Z New outbound peer connected: version: 70016, blocks=2140571, peer=493, peeraddr=65.21.56.203:37070 (full-relay)
2023-03-17T10:07:40Z New outbound peer connected: version: 70016, blocks=1443012, peer=492, peeraddr=31.211.105.232:37070 (full-relay)
2023-03-17T10:07:41Z New outbound peer connected: version: 70016, blocks=2140570, peer=490, peeraddr=81.169.156.24:37070 (full-relay)
2023-03-17T10:07:51Z New outbound peer connected: version: 70016, blocks=2140571, peer=496, peeraddr=80.203.27.91:37070 (block-relay)
2023-03-17T10:07:52Z New outbound peer connected: version: 70016, blocks=2140571, peer=497, peeraddr=172.83.108.39:37070 (block-relay)
2023-03-17T10:08:09Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:08:09Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:08:10Z Disconnecting and discouraging peer 485!
2023-03-17T10:08:11Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:08:11Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:08:11Z Disconnecting and discouraging peer 490!
2023-03-17T10:08:11Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:08:11Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:08:11Z Disconnecting and discouraging peer 491!
2023-03-17T10:08:12Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:08:12Z Disconnecting and discouraging peer 493!
2023-03-17T10:08:13Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:08:13Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:08:13Z Disconnecting and discouraging peer 496!
2023-03-17T10:08:14Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:08:14Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:08:14Z Disconnecting and discouraging peer 497!
2023-03-17T10:08:33Z New outbound peer connected: version: 70016, blocks=2061157, peer=499, peeraddr=178.217.159.221:37070 (full-relay)
2023-03-17T10:08:39Z New outbound peer connected: version: 70016, blocks=2140572, peer=500, peeraddr=78.29.45.23:37070 (full-relay)
2023-03-17T10:08:40Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:08:40Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:08:40Z Disconnecting and discouraging peer 500!
2023-03-17T10:08:51Z New outbound peer connected: version: 70016, blocks=2140572, peer=501, peeraddr=88.216.164.30:37070 (full-relay)
2023-03-17T10:08:51Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:08:51Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:08:51Z Disconnecting and discouraging peer 501!
2023-03-17T10:08:58Z Function `GetContentFromAddressSubscriptions` failed with execute timeout
2023-03-17T10:08:58Z ErrorLogCallback: 9; Message: statement aborts at 50: [
2023-03-17T10:09:09Z New outbound peer connected: version: 70016, blocks=2140572, peer=502, peeraddr=38.23.148.12:37070 (full-relay)
2023-03-17T10:09:09Z New outbound peer connected: version: 70016, blocks=2140572, peer=503, peeraddr=142.196.98.89:37070 (full-relay)
2023-03-17T10:09:10Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:09:10Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:09:10Z Disconnecting and discouraging peer 502!
2023-03-17T10:09:12Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:09:12Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:09:12Z Disconnecting and discouraging peer 503!
2023-03-17T10:09:26Z New outbound peer connected: version: 70016, blocks=2140572, peer=504, peeraddr=81.169.211.179:37070 (full-relay)
2023-03-17T10:09:27Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:09:27Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:09:27Z Disconnecting and discouraging peer 504!
2023-03-17T10:09:33Z New outbound peer connected: version: 70016, blocks=2140572, peer=506, peeraddr=84.201.247.70:37070 (full-relay)
2023-03-17T10:09:34Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:09:34Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:09:34Z Disconnecting and discouraging peer 506!
2023-03-17T10:09:35Z New outbound peer connected: version: 70016, blocks=2140572, peer=507, peeraddr=172.83.108.40:37070 (full-relay)
2023-03-17T10:09:36Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:09:36Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:09:36Z Disconnecting and discouraging peer 507!
2023-03-17T10:09:51Z Function `GetAccountProfiles` failed with execute timeout
2023-03-17T10:09:51Z ErrorLogCallback: 9; Message: statement aborts at 0: [
2023-03-17T10:09:53Z New outbound peer connected: version: 70016, blocks=2140573, peer=508, peeraddr=142.116.40.18:37070 (full-relay)
2023-03-17T10:09:54Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:09:54Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:09:54Z Disconnecting and discouraging peer 508!
2023-03-17T10:09:59Z New outbound peer connected: version: 70016, blocks=2140573, peer=509, peeraddr=172.83.108.37:37070 (full-relay)
2023-03-17T10:10:01Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:10:01Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:10:01Z Disconnecting and discouraging peer 509!
2023-03-17T10:10:25Z New outbound peer connected: version: 70016, blocks=2140574, peer=510, peeraddr=2.94.16.129:37070 (full-relay)
2023-03-17T10:10:42Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:10:42Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:10:58Z Disconnecting and discouraging peer 510!
2023-03-17T10:10:59Z New outbound peer connected: version: 70016, blocks=2140574, peer=511, peeraddr=109.195.147.45:37070 (full-relay)
2023-03-17T10:10:59Z New outbound peer connected: version: 70016, blocks=2140574, peer=512, peeraddr=80.239.36.24:37070 (full-relay)
2023-03-17T10:11:05Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:11:05Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:11:21Z Disconnecting and discouraging peer 511!
2023-03-17T10:11:21Z New outbound peer connected: version: 70016, blocks=2140574, peer=513, peeraddr=202.61.253.55:37070 (full-relay)
2023-03-17T10:11:26Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:11:26Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:11:43Z Disconnecting and discouraging peer 512!
2023-03-17T10:11:54Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:11:54Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:12:11Z Disconnecting and discouraging peer 513!
2023-03-17T10:12:11Z New outbound peer connected: version: 70016, blocks=2140575, peer=514, peeraddr=18.237.17.181:37070 (full-relay)
2023-03-17T10:12:21Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:12:21Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:12:39Z Disconnecting and discouraging peer 514!
2023-03-17T10:15:08Z New outbound peer connected: version: 70016, blocks=2140575, peer=515, peeraddr=172.83.108.41:37070 (full-relay)
2023-03-17T10:15:21Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:15:21Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:15:42Z Disconnecting and discouraging peer 515!
2023-03-17T10:15:42Z Potential stale tip detected, will try using extra outbound peer (last tip update: 183 seconds ago)
2023-03-17T10:18:50Z New outbound peer connected: version: 70016, blocks=2140578, peer=517, peeraddr=188.243.168.217:37070 (full-relay)
2023-03-17T10:19:03Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:19:03Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:19:21Z Disconnecting and discouraging peer 517!
2023-03-17T10:20:13Z New outbound peer connected: version: 70015, blocks=2139949, peer=519, peeraddr=77.173.8.2:37070 (full-relay)
2023-03-17T10:22:57Z New outbound peer connected: version: 70016, blocks=2061157, peer=520, peeraddr=178.217.159.221:37070 (full-relay)
2023-03-17T10:23:52Z New outbound peer connected: version: 70016, blocks=2140580, peer=521, peeraddr=75.156.77.38:37070 (full-relay)
2023-03-17T10:24:16Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:24:16Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:24:37Z Disconnecting and discouraging peer 521!
2023-03-17T10:26:39Z New outbound peer connected: version: 70016, blocks=2136925, peer=524, peeraddr=209.145.50.185:37070 (full-relay)
2023-03-17T10:28:07Z Disconnecting outbound peer 480 for old chain, best known block = <none>
2023-03-17T10:29:02Z Disconnecting outbound peer 488 for old chain, best known block = <none>
2023-03-17T10:29:39Z Function `GetHistoricalFeed` failed with execute timeout
2023-03-17T10:29:39Z ErrorLogCallback: 9; Message: statement aborts at 12: [
2023-03-17T10:29:42Z Disconnecting outbound peer 492 for old chain, best known block = <none>
2023-03-17T10:32:32Z New outbound peer connected: version: 70016, blocks=1444556, peer=530, peeraddr=31.211.105.232:37070 (full-relay)
2023-03-17T10:33:30Z New outbound peer connected: version: 70016, blocks=2133670, peer=531, peeraddr=209.55.111.169:37070 (full-relay)
2023-03-17T10:34:51Z New outbound peer connected: version: 70016, blocks=2126978, peer=533, peeraddr=185.183.197.82:37070 (full-relay)
2023-03-17T10:37:16Z New outbound peer connected: version: 70015, blocks=2139965, peer=537, peeraddr=77.173.8.2:37070 (full-relay)
2023-03-17T10:39:37Z New outbound peer connected: version: 70016, blocks=2061157, peer=539, peeraddr=178.217.159.221:37070 (full-relay)
2023-03-17T10:44:22Z Function `GetAccountProfiles` failed with execute timeout
2023-03-17T10:44:22Z ErrorLogCallback: 9; Message: statement aborts at 788: [
2023-03-17T10:46:41Z socket receive timeout: 1201s
2023-03-17T10:47:08Z Potential stale tip detected, will try using extra outbound peer (last tip update: 648 seconds ago)
2023-03-17T10:50:36Z New outbound peer connected: version: 70015, blocks=2139981, peer=546, peeraddr=77.173.8.2:37070 (full-relay)
2023-03-17T10:52:20Z Leaving block file 89: CBlockFileInfo(blocks=10520, size=134208252, heights=2130599...2140576, time=2023-03-10...2023-03-17)
2023-03-17T10:54:34Z Disconnecting outbound peer 530 for old chain, best known block = <none>
2023-03-17T10:55:20Z New outbound peer connected: version: 70016, blocks=1446086, peer=547, peeraddr=31.211.105.232:37070 (full-relay)
2023-03-17T10:55:32Z Disconnecting outbound peer 531 for old chain, best known block = <none>
2023-03-17T10:56:53Z Disconnecting outbound peer 533 for old chain, best known block = <none>
2023-03-17T10:57:16Z New outbound peer connected: version: 70016, blocks=2126978, peer=548, peeraddr=185.183.197.82:37070 (full-relay)
2023-03-17T10:57:23Z New outbound peer connected: version: 70016, blocks=2133670, peer=549, peeraddr=209.55.111.169:37070 (full-relay)
2023-03-17T10:58:14Z New outbound peer connected: version: 70016, blocks=2061157, peer=550, peeraddr=178.217.159.221:37070 (full-relay)
2023-03-17T11:02:14Z New outbound peer connected: version: 70015, blocks=2139988, peer=551, peeraddr=77.173.8.2:37070 (full-relay)
2023-03-17T11:08:11Z Potential stale tip detected, will try using extra outbound peer (last tip update: 664 seconds ago)
2023-03-17T11:13:17Z New outbound peer connected: version: 70016, blocks=2126978, peer=553, peeraddr=185.183.197.82:37070 (full-relay)
2023-03-17T11:17:22Z Disconnecting outbound peer 547 for old chain, best known block = <none>
2023-03-17T11:18:23Z New outbound peer connected: version: 70015, blocks=2140002, peer=555, peeraddr=77.173.8.2:37070 (full-relay)
2023-03-17T11:19:25Z Disconnecting outbound peer 549 for old chain, best known block = <none>
2023-03-17T11:20:16Z Disconnecting outbound peer 550 for old chain, best known block = <none>
2023-03-17T11:23:26Z New outbound peer connected: version: 70016, blocks=1448259, peer=557, peeraddr=31.211.105.232:37070 (full-relay)
2023-03-17T11:26:07Z New outbound peer connected: version: 70016, blocks=2061157, peer=558, peeraddr=178.217.159.221:37070 (full-relay)
2023-03-17T11:27:05Z New outbound peer connected: version: 70016, blocks=2133670, peer=559, peeraddr=209.55.111.169:37070 (full-relay)
2023-03-17T11:30:42Z New outbound peer connected: version: 70015, blocks=2140016, peer=560, peeraddr=77.173.8.2:37070 (full-relay)
2023-03-17T11:35:19Z Disconnecting outbound peer 553 for old chain, best known block = <none>
2023-03-17T11:37:32Z New outbound peer connected: version: 70016, blocks=2061157, peer=563, peeraddr=178.217.159.221:37070 (full-relay)
2023-03-17T11:40:10Z New outbound peer connected: version: 70016, blocks=2126978, peer=564, peeraddr=185.183.197.82:37070 (full-relay)
2023-03-17T11:41:16Z New outbound peer connected: version: 70015, blocks=2140035, peer=565, peeraddr=77.173.8.2:37070 (full-relay)
2023-03-17T11:45:28Z Disconnecting outbound peer 557 for old chain, best known block = <none>
2023-03-17T11:46:50Z New outbound peer connected: version: 70016, blocks=1450545, peer=566, peeraddr=31.211.105.232:37070 (full-relay)
2023-03-17T11:49:07Z Disconnecting outbound peer 559 for old chain, best known block = <none>
2023-03-17T11:49:23Z New outbound peer connected: version: 70016, blocks=2061157, peer=569, peeraddr=178.217.159.221:37070 (full-relay)
2023-03-17T11:50:15Z Potential stale tip detected, will try using extra outbound peer (last tip update: 328 seconds ago)
2023-03-17T11:53:14Z New outbound peer connected: version: 70016, blocks=2126978, peer=571, peeraddr=185.183.197.82:37070 (full-relay)
2023-03-17T11:58:52Z New outbound peer connected: version: 70016, blocks=2133670, peer=578, peeraddr=209.55.111.169:37070 (full-relay)
2023-03-17T11:59:42Z New outbound peer connected: version: 70015, blocks=2140050, peer=579, peeraddr=77.173.8.2:37070 (full-relay)
2023-03-17T12:05:16Z New outbound peer connected: version: 70016, blocks=2126978, peer=580, peeraddr=185.183.197.82:37070 (full-relay)
2023-03-17T12:06:41Z New outbound peer connected: version: 70016, blocks=2061157, peer=585, peeraddr=178.217.159.221:37070 (full-relay)
2023-03-17T12:08:52Z Disconnecting outbound peer 566 for old chain, best known block = <none>
2023-03-17T12:09:52Z New outbound peer connected: version: 70016, blocks=1452393, peer=587, peeraddr=31.211.105.232:37070 (full-relay)
2023-03-17T12:10:37Z New outbound peer connected: version: 70015, blocks=2140067, peer=588, peeraddr=77.173.8.2:37070 (full-relay)
2023-03-17T12:11:41Z Function `GetAccountProfiles` failed with execute timeout
2023-03-17T12:11:41Z ErrorLogCallback: 9; Message: statement aborts at 634: [
2023-03-17T12:12:41Z New outbound peer connected: version: 70016, blocks=2140591, peer=589, peeraddr=65.21.57.14:37070 (full-relay)
2023-03-17T12:16:51Z New outbound peer connected: version: 70016, blocks=2126978, peer=592, peeraddr=185.183.197.82:37070 (full-relay)
2023-03-17T12:17:50Z New outbound peer connected: version: 70016, blocks=2140614, peer=593, peeraddr=89.179.78.57:37070 (full-relay)
2023-03-17T12:20:54Z Disconnecting outbound peer 578 for old chain, best known block = <none>
2023-03-17T12:23:12Z New outbound peer connected: version: 70015, blocks=2140074, peer=597, peeraddr=77.173.8.2:37070 (full-relay)
2023-03-17T12:25:25Z New outbound peer connected: version: 70016, blocks=2061157, peer=598, peeraddr=178.217.159.221:37070 (full-relay)
2023-03-17T12:26:49Z New outbound peer connected: version: 70016, blocks=2133670, peer=601, peeraddr=209.55.111.169:37070 (full-relay)
2023-03-17T12:28:32Z New outbound peer connected: version: 70016, blocks=2126978, peer=603, peeraddr=185.183.197.82:37070 (full-relay)
2023-03-17T12:31:47Z Potential stale tip detected, will try using extra outbound peer (last tip update: 207 seconds ago)
2023-03-17T12:31:54Z Disconnecting outbound peer 587 for old chain, best known block = <none>
2023-03-17T12:33:42Z New outbound peer connected: version: 70015, blocks=2140086, peer=609, peeraddr=77.173.8.2:37070 (full-relay)
2023-03-17T12:34:43Z Disconnecting outbound peer 589 for old chain, best known block = <none>
2023-03-17T12:35:10Z New outbound peer connected: version: 70016, blocks=1454728, peer=610, peeraddr=31.211.105.232:37070 (full-relay)
2023-03-17T12:35:51Z New outbound peer connected: version: 70016, blocks=2061157, peer=611, peeraddr=178.217.159.221:37070 (full-relay)
2023-03-17T12:44:02Z New outbound peer connected: version: 70015, blocks=2140097, peer=615, peeraddr=77.173.8.2:37070 (full-relay)

In banner.log there were following lines regarding old nodes (time in UTC+3):

2023-03-16 13:25:52 === Old node version banned (0.20.29):
2023-03-16 13:25:52 178.209.98.206:12362        /Satoshi:0.20.29/       2019352 -1      -1      328     7711    2023-03-16 13:25:02     inbound
2023-03-17 18:05:48 === Old node version banned (0.20.29):
2023-03-17 18:05:48 77.173.8.2:37070    /Satoshi:0.20.29/       2140234 2077186 -1      165503  35575   2023-03-17 18:05:46     outbound-full-relay
andyoknen commented 1 year ago

What is the date of the first record in debug.log?

HiHat commented 1 year ago

Near about 2023-03-16T21:50:00Z. Can't attach debug.log as a file.

Seems that my node was forked by 77.173.8.2 peer with 0.20.29 version.

andyoknen commented 1 year ago

The main catch is that the knot can fork because of one other node, but for no good reason it cannot stay in this fork for a long time, ignoring the main chain

HiHat commented 1 year ago

Found connected node list dumped about 2023-03-16T23:06:35Z:

$ pocketcoin-cli getpeerinfo | jq -r '.[] | [.addr, .subver, .pingtime, .startingheight, .synced_headers, .synced_blocks, .bytesrecv, .bytessent, (.conntime|todate), .timeoffset, .connection_type] | @tsv' | column -t
95.31.45.162:37070     /Satoshi:0.21.1/  0.012638  2139830  2139929  2139929  445910   9142    2023-03-16T21:58:53Z  0   block-relay-only
91.206.15.176:37070    /Satoshi:0.21.1/  0.010894  2139830  2139931  2139931  952776   17250   2023-03-16T21:58:54Z  -1  block-relay-only
73.96.245.93:37070     /Satoshi:0.21.1/  0.201046  2139830  2139929  2139929  691146   251659  2023-03-16T21:59:16Z  1   outbound-full-relay
67.140.183.89:37070    /Satoshi:0.21.2/  0.129884  2139831  2139929  2139929  498847   94594   2023-03-16T21:59:24Z  1   outbound-full-relay
135.181.196.243:37070  /Satoshi:0.21.2/  0.017554  2139831  2139931  2139931  551241   419525  2023-03-16T21:59:34Z  2   outbound-full-relay
94.190.60.151:37070    /Satoshi:0.21.2/  0.038438  2139831  2139931  2139931  312999   107954  2023-03-16T21:59:35Z  1   outbound-full-relay
162.246.52.155:37070   /Satoshi:0.21.1/  0.132026  2139831  2139931  2139931  571476   96049   2023-03-16T21:59:35Z  1   outbound-full-relay
85.236.190.129:37070   /Satoshi:0.21.1/  0.022291  2139831  2139931  2139931  1216752  95616   2023-03-16T21:59:54Z  -1  outbound-full-relay
68.118.120.152:37070   /Satoshi:0.21.1/  0.161434  2139832  2139929  2139929  161682   107349  2023-03-16T22:00:06Z  2   outbound-full-relay
185.9.187.123:54132    /Satoshi:0.21.2/  0.018141  2139871  2139930  2139930  462779   90495   2023-03-16T22:30:16Z  2   inbound
37.48.87.107:37070     /Satoshi:0.21.2/  0.042353  2139903  2139931  2139931  57315    20569   2023-03-16T23:06:35Z  1   outbound-full-relay

77.173.8.2 was not connected and therefore not banned by banner script! It was connected and banned only at 2023-03-17 18:05. I thought that one of 0.21 connected node was forked and then poisoned my node.

andyoknen commented 1 year ago

I am inclined to the fact that there are several nodes in the fork in the network. I have no idea how to return all nodes to the main chain - this situation is normal and should be resolved over time

andyoknen commented 8 months ago

In version 0.22, an optional disconnectold argument has been added to disable old nodes or nodes in the fork - this should solve some of the problems with outdated nodes and the potential forks they cause.