Closed HiHat closed 8 months ago
We are dealing with the problem
When you do a bug report like this in future. I highly suggest giving a list of the connected nodes. Old versions of nodes can influence and cause this issues (and have done in the past).
Where there any versions of the node software lower than V21 connected at the time?
@AOK can you give us an update of what's possibly causing this because lots of node operators are getting this or similar at the moment. An update in the node chat would be good.
When you do a bug report like this in future. I highly suggest giving a list of the connected nodes.
According to provided report and log there were no connections as it was cold start from forked checkpoint (at height 2140978) obtained from snapshot.pocketnet.app! If we see at first line of log:
2023-03-18T21:35:33Z New outbound peer connected: version: 70016, blocks=2142704, peer=0, peeraddr=202.61.253.55:37070 (full-relay)
Then we found that first connection (due to peer=0) was made to 202.61.253.55 peer with strange 70016 version! Unfortunately node software versions (like 0.21.1) not dumped to logs! Currently 202.61.253.55 peer hosted 0.21.1 version.
If we analyze log further then we found that fork was detected and node began rollback to 2140568 best block. Unfortunately 2140569 block was not correctly validated and new fork began:
2023-03-18T22:02:36Z Warning: SocialConsensus type:305 validate tx:0597aa5dc60e70a16d497b89d58ef337749f4cd751ba967e232099d905c35bdc blk:aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed with result:23 at height:2140569
2023-03-18T22:02:36Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-18T22:02:36Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-18T22:02:47Z Block validated: 2140569 BH: 7aec19f7c5f7673fe46c3ba56ec8d3ed0ab7bd3da4354d62b11c8a48231b97a9
2023-03-18T22:03:04Z Block indexed: 2140569 BH: 7aec19f7c5f7673fe46c3ba56ec8d3ed0ab7bd3da4354d62b11c8a48231b97a9
2023-03-18T22:03:04Z +++ Block connected to chain: 2140569 BH: 7aec19f7c5f7673fe46c3ba56ec8d3ed0ab7bd3da4354d62b11c8a48231b97a9
At the moment, together with @lostystyg, we are studying this problem, although at our stand the error is different - number 12, which indicates that there is no necessary data to connect the block from the main chain and (here's the problem) goes into fork again.
I can share forked checkpoint (from 2023-03-18) to help reproduce error number 23!
We have one copy for analysis, but please leave yours just in case. After yesterday's study, we have so far come to the preliminary conclusion that the forks are related to the late update of some nodes, as some releases include consensus changes (adding new types of transactions or changing their behavior). Maybe you still have a log file that shows the version of the node starting from 0.20.29?
I have 2 log files with node forking (banner.log and debug.log). Node forked at 2023-03-17 10:06 UTC.
In debug.log was following lines about forking (time in UTC):
2023-03-16T21:58:53Z New outbound peer connected: version: 70016, blocks=2139830, peer=0, peeraddr=95.31.45.162:37070 (block-relay)
2023-03-16T21:58:53Z Leaving InitialBlockDownload (latching to false)
2023-03-16T21:59:24Z New outbound peer connected: version: 70016, blocks=2139830, peer=1, peeraddr=91.206.15.176:37070 (block-relay)
2023-03-16T21:59:27Z New outbound peer connected: version: 70016, blocks=2139830, peer=2, peeraddr=73.96.245.93:37070 (full-relay)
2023-03-16T21:59:29Z New outbound peer connected: version: 70016, blocks=2139830, peer=3, peeraddr=81.169.156.24:37070 (full-relay)
2023-03-16T21:59:32Z New outbound peer connected: version: 70016, blocks=2139831, peer=4, peeraddr=67.140.183.89:37070 (full-relay)
2023-03-16T21:59:34Z New outbound peer connected: version: 70016, blocks=2139831, peer=5, peeraddr=135.181.196.243:37070 (full-relay)
2023-03-16T21:59:35Z New outbound peer connected: version: 70016, blocks=2139831, peer=6, peeraddr=94.190.60.151:37070 (full-relay)
2023-03-16T21:59:36Z New outbound peer connected: version: 70016, blocks=2139831, peer=7, peeraddr=162.246.52.155:37070 (full-relay)
2023-03-16T21:59:54Z New outbound peer connected: version: 70016, blocks=2139831, peer=9, peeraddr=85.236.190.129:37070 (full-relay)
2023-03-16T22:00:06Z New outbound peer connected: version: 70016, blocks=2139832, peer=10, peeraddr=68.118.120.152:37070 (full-relay)
2023-03-16T22:21:28Z New outbound peer connected: version: 70016, blocks=2139858, peer=23, peeraddr=81.169.156.24:37070 (full-relay)
...
2023-03-16T22:51:31Z New outbound peer connected: version: 70016, blocks=2139888, peer=45, peeraddr=109.195.147.45:37070 (full-relay)
2023-03-16T23:06:33Z New outbound peer connected: version: 70015, blocks=2139276, peer=57, peeraddr=77.173.8.2:37070 (full-relay)
2023-03-16T23:06:35Z New outbound peer connected: version: 70016, blocks=2139903, peer=58, peeraddr=37.48.87.107:37070 (full-relay)
...
2023-03-17T02:27:30Z New outbound peer connected: version: 70016, blocks=2140114, peer=197, peeraddr=188.18.52.142:37070 (full-relay)
...
2023-03-17T04:14:20Z New outbound peer connected: version: 70016, blocks=2140223, peer=262, peeraddr=65.21.252.135:37070 (full-relay)
...
2023-03-17T05:20:49Z Potential stale tip detected, will try using extra outbound peer (last tip update: 366 seconds ago)
2023-03-17T05:20:55Z New outbound peer connected: version: 70016, blocks=2140278, peer=310, peeraddr=142.116.40.18:37070 (full-relay)
2023-03-17T05:27:32Z New outbound peer connected: version: 70016, blocks=2140284, peer=316, peeraddr=140.99.153.138:37070 (full-relay)
...
2023-03-17T10:06:04Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:06:04Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:06:04Z Disconnecting and discouraging peer 316!
2023-03-17T10:06:05Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:06:05Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:06:05Z Disconnecting and discouraging peer 0!
2023-03-17T10:06:05Z New outbound peer connected: version: 70016, blocks=2133670, peer=480, peeraddr=209.55.111.169:37070 (full-relay)
2023-03-17T10:06:05Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:06:05Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:06:05Z Disconnecting and discouraging peer 1!
2023-03-17T10:06:08Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:06:08Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:06:08Z Disconnecting and discouraging peer 7!
2023-03-17T10:06:09Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:06:09Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:06:09Z Disconnecting and discouraging peer 32!
2023-03-17T10:06:09Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:06:09Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:06:09Z Disconnecting and discouraging peer 111!
2023-03-17T10:06:13Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:06:13Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:06:13Z Disconnecting and discouraging peer 315!
2023-03-17T10:06:14Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:06:14Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:06:14Z Disconnecting and discouraging peer 4!
2023-03-17T10:06:15Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:06:15Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:06:15Z Disconnecting and discouraging peer 6!
2023-03-17T10:06:15Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:06:15Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:06:15Z Disconnecting and discouraging peer 9!
2023-03-17T10:06:16Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:06:16Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:06:16Z Disconnecting and discouraging peer 58!
2023-03-17T10:06:17Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:06:17Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:06:17Z Disconnecting and discouraging peer 197!
2023-03-17T10:06:17Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:06:17Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:06:17Z Disconnecting and discouraging peer 262!
2023-03-17T10:06:19Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:06:19Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:06:19Z Disconnecting and discouraging peer 397!
2023-03-17T10:06:29Z New outbound peer connected: version: 70016, blocks=2140569, peer=484, peeraddr=135.181.196.243:37070 (full-relay)
2023-03-17T10:06:29Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:06:29Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:06:29Z Disconnecting and discouraging peer 484!
2023-03-17T10:06:30Z New outbound peer connected: version: 70016, blocks=2140568, peer=486, peeraddr=89.106.125.173:37070 (full-relay)
2023-03-17T10:07:00Z New outbound peer connected: version: 70016, blocks=2126978, peer=488, peeraddr=185.183.197.82:37070 (full-relay)
2023-03-17T10:07:07Z New outbound peer connected: version: 70016, blocks=2140568, peer=485, peeraddr=83.151.14.174:37070 (full-relay)
2023-03-17T10:07:12Z New outbound peer connected: version: 70016, blocks=2140570, peer=489, peeraddr=172.83.108.38:37070 (full-relay)
2023-03-17T10:07:14Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:07:14Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:07:14Z Disconnecting and discouraging peer 489!
2023-03-17T10:07:32Z New outbound peer connected: version: 70016, blocks=2140571, peer=491, peeraddr=109.197.196.106:37070 (full-relay)
2023-03-17T10:07:34Z New outbound peer connected: version: 70016, blocks=2140571, peer=493, peeraddr=65.21.56.203:37070 (full-relay)
2023-03-17T10:07:40Z New outbound peer connected: version: 70016, blocks=1443012, peer=492, peeraddr=31.211.105.232:37070 (full-relay)
2023-03-17T10:07:41Z New outbound peer connected: version: 70016, blocks=2140570, peer=490, peeraddr=81.169.156.24:37070 (full-relay)
2023-03-17T10:07:51Z New outbound peer connected: version: 70016, blocks=2140571, peer=496, peeraddr=80.203.27.91:37070 (block-relay)
2023-03-17T10:07:52Z New outbound peer connected: version: 70016, blocks=2140571, peer=497, peeraddr=172.83.108.39:37070 (block-relay)
2023-03-17T10:08:09Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:08:09Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:08:10Z Disconnecting and discouraging peer 485!
2023-03-17T10:08:11Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:08:11Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:08:11Z Disconnecting and discouraging peer 490!
2023-03-17T10:08:11Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:08:11Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:08:11Z Disconnecting and discouraging peer 491!
2023-03-17T10:08:12Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:08:12Z Disconnecting and discouraging peer 493!
2023-03-17T10:08:13Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:08:13Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:08:13Z Disconnecting and discouraging peer 496!
2023-03-17T10:08:14Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:08:14Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:08:14Z Disconnecting and discouraging peer 497!
2023-03-17T10:08:33Z New outbound peer connected: version: 70016, blocks=2061157, peer=499, peeraddr=178.217.159.221:37070 (full-relay)
2023-03-17T10:08:39Z New outbound peer connected: version: 70016, blocks=2140572, peer=500, peeraddr=78.29.45.23:37070 (full-relay)
2023-03-17T10:08:40Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:08:40Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:08:40Z Disconnecting and discouraging peer 500!
2023-03-17T10:08:51Z New outbound peer connected: version: 70016, blocks=2140572, peer=501, peeraddr=88.216.164.30:37070 (full-relay)
2023-03-17T10:08:51Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:08:51Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:08:51Z Disconnecting and discouraging peer 501!
2023-03-17T10:08:58Z Function `GetContentFromAddressSubscriptions` failed with execute timeout
2023-03-17T10:08:58Z ErrorLogCallback: 9; Message: statement aborts at 50: [
2023-03-17T10:09:09Z New outbound peer connected: version: 70016, blocks=2140572, peer=502, peeraddr=38.23.148.12:37070 (full-relay)
2023-03-17T10:09:09Z New outbound peer connected: version: 70016, blocks=2140572, peer=503, peeraddr=142.196.98.89:37070 (full-relay)
2023-03-17T10:09:10Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:09:10Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:09:10Z Disconnecting and discouraging peer 502!
2023-03-17T10:09:12Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:09:12Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:09:12Z Disconnecting and discouraging peer 503!
2023-03-17T10:09:26Z New outbound peer connected: version: 70016, blocks=2140572, peer=504, peeraddr=81.169.211.179:37070 (full-relay)
2023-03-17T10:09:27Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:09:27Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:09:27Z Disconnecting and discouraging peer 504!
2023-03-17T10:09:33Z New outbound peer connected: version: 70016, blocks=2140572, peer=506, peeraddr=84.201.247.70:37070 (full-relay)
2023-03-17T10:09:34Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:09:34Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:09:34Z Disconnecting and discouraging peer 506!
2023-03-17T10:09:35Z New outbound peer connected: version: 70016, blocks=2140572, peer=507, peeraddr=172.83.108.40:37070 (full-relay)
2023-03-17T10:09:36Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:09:36Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:09:36Z Disconnecting and discouraging peer 507!
2023-03-17T10:09:51Z Function `GetAccountProfiles` failed with execute timeout
2023-03-17T10:09:51Z ErrorLogCallback: 9; Message: statement aborts at 0: [
2023-03-17T10:09:53Z New outbound peer connected: version: 70016, blocks=2140573, peer=508, peeraddr=142.116.40.18:37070 (full-relay)
2023-03-17T10:09:54Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:09:54Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:09:54Z Disconnecting and discouraging peer 508!
2023-03-17T10:09:59Z New outbound peer connected: version: 70016, blocks=2140573, peer=509, peeraddr=172.83.108.37:37070 (full-relay)
2023-03-17T10:10:01Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:10:01Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:10:01Z Disconnecting and discouraging peer 509!
2023-03-17T10:10:25Z New outbound peer connected: version: 70016, blocks=2140574, peer=510, peeraddr=2.94.16.129:37070 (full-relay)
2023-03-17T10:10:42Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:10:42Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:10:58Z Disconnecting and discouraging peer 510!
2023-03-17T10:10:59Z New outbound peer connected: version: 70016, blocks=2140574, peer=511, peeraddr=109.195.147.45:37070 (full-relay)
2023-03-17T10:10:59Z New outbound peer connected: version: 70016, blocks=2140574, peer=512, peeraddr=80.239.36.24:37070 (full-relay)
2023-03-17T10:11:05Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:11:05Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:11:21Z Disconnecting and discouraging peer 511!
2023-03-17T10:11:21Z New outbound peer connected: version: 70016, blocks=2140574, peer=513, peeraddr=202.61.253.55:37070 (full-relay)
2023-03-17T10:11:26Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:11:26Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:11:43Z Disconnecting and discouraging peer 512!
2023-03-17T10:11:54Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:11:54Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:12:11Z Disconnecting and discouraging peer 513!
2023-03-17T10:12:11Z New outbound peer connected: version: 70016, blocks=2140575, peer=514, peeraddr=18.237.17.181:37070 (full-relay)
2023-03-17T10:12:21Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:12:21Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:12:39Z Disconnecting and discouraging peer 514!
2023-03-17T10:15:08Z New outbound peer connected: version: 70016, blocks=2140575, peer=515, peeraddr=172.83.108.41:37070 (full-relay)
2023-03-17T10:15:21Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:15:21Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:15:42Z Disconnecting and discouraging peer 515!
2023-03-17T10:15:42Z Potential stale tip detected, will try using extra outbound peer (last tip update: 183 seconds ago)
2023-03-17T10:18:50Z New outbound peer connected: version: 70016, blocks=2140578, peer=517, peeraddr=188.243.168.217:37070 (full-relay)
2023-03-17T10:19:03Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:19:03Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:19:21Z Disconnecting and discouraging peer 517!
2023-03-17T10:20:13Z New outbound peer connected: version: 70015, blocks=2139949, peer=519, peeraddr=77.173.8.2:37070 (full-relay)
2023-03-17T10:22:57Z New outbound peer connected: version: 70016, blocks=2061157, peer=520, peeraddr=178.217.159.221:37070 (full-relay)
2023-03-17T10:23:52Z New outbound peer connected: version: 70016, blocks=2140580, peer=521, peeraddr=75.156.77.38:37070 (full-relay)
2023-03-17T10:24:16Z WARNING: SocialConsensus validating failed with result 23 for block aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592
2023-03-17T10:24:16Z ERROR: ConnectTip: ConnectBlock aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592 failed, failed-validate-social-consensus
2023-03-17T10:24:37Z Disconnecting and discouraging peer 521!
2023-03-17T10:26:39Z New outbound peer connected: version: 70016, blocks=2136925, peer=524, peeraddr=209.145.50.185:37070 (full-relay)
2023-03-17T10:28:07Z Disconnecting outbound peer 480 for old chain, best known block = <none>
2023-03-17T10:29:02Z Disconnecting outbound peer 488 for old chain, best known block = <none>
2023-03-17T10:29:39Z Function `GetHistoricalFeed` failed with execute timeout
2023-03-17T10:29:39Z ErrorLogCallback: 9; Message: statement aborts at 12: [
2023-03-17T10:29:42Z Disconnecting outbound peer 492 for old chain, best known block = <none>
2023-03-17T10:32:32Z New outbound peer connected: version: 70016, blocks=1444556, peer=530, peeraddr=31.211.105.232:37070 (full-relay)
2023-03-17T10:33:30Z New outbound peer connected: version: 70016, blocks=2133670, peer=531, peeraddr=209.55.111.169:37070 (full-relay)
2023-03-17T10:34:51Z New outbound peer connected: version: 70016, blocks=2126978, peer=533, peeraddr=185.183.197.82:37070 (full-relay)
2023-03-17T10:37:16Z New outbound peer connected: version: 70015, blocks=2139965, peer=537, peeraddr=77.173.8.2:37070 (full-relay)
2023-03-17T10:39:37Z New outbound peer connected: version: 70016, blocks=2061157, peer=539, peeraddr=178.217.159.221:37070 (full-relay)
2023-03-17T10:44:22Z Function `GetAccountProfiles` failed with execute timeout
2023-03-17T10:44:22Z ErrorLogCallback: 9; Message: statement aborts at 788: [
2023-03-17T10:46:41Z socket receive timeout: 1201s
2023-03-17T10:47:08Z Potential stale tip detected, will try using extra outbound peer (last tip update: 648 seconds ago)
2023-03-17T10:50:36Z New outbound peer connected: version: 70015, blocks=2139981, peer=546, peeraddr=77.173.8.2:37070 (full-relay)
2023-03-17T10:52:20Z Leaving block file 89: CBlockFileInfo(blocks=10520, size=134208252, heights=2130599...2140576, time=2023-03-10...2023-03-17)
2023-03-17T10:54:34Z Disconnecting outbound peer 530 for old chain, best known block = <none>
2023-03-17T10:55:20Z New outbound peer connected: version: 70016, blocks=1446086, peer=547, peeraddr=31.211.105.232:37070 (full-relay)
2023-03-17T10:55:32Z Disconnecting outbound peer 531 for old chain, best known block = <none>
2023-03-17T10:56:53Z Disconnecting outbound peer 533 for old chain, best known block = <none>
2023-03-17T10:57:16Z New outbound peer connected: version: 70016, blocks=2126978, peer=548, peeraddr=185.183.197.82:37070 (full-relay)
2023-03-17T10:57:23Z New outbound peer connected: version: 70016, blocks=2133670, peer=549, peeraddr=209.55.111.169:37070 (full-relay)
2023-03-17T10:58:14Z New outbound peer connected: version: 70016, blocks=2061157, peer=550, peeraddr=178.217.159.221:37070 (full-relay)
2023-03-17T11:02:14Z New outbound peer connected: version: 70015, blocks=2139988, peer=551, peeraddr=77.173.8.2:37070 (full-relay)
2023-03-17T11:08:11Z Potential stale tip detected, will try using extra outbound peer (last tip update: 664 seconds ago)
2023-03-17T11:13:17Z New outbound peer connected: version: 70016, blocks=2126978, peer=553, peeraddr=185.183.197.82:37070 (full-relay)
2023-03-17T11:17:22Z Disconnecting outbound peer 547 for old chain, best known block = <none>
2023-03-17T11:18:23Z New outbound peer connected: version: 70015, blocks=2140002, peer=555, peeraddr=77.173.8.2:37070 (full-relay)
2023-03-17T11:19:25Z Disconnecting outbound peer 549 for old chain, best known block = <none>
2023-03-17T11:20:16Z Disconnecting outbound peer 550 for old chain, best known block = <none>
2023-03-17T11:23:26Z New outbound peer connected: version: 70016, blocks=1448259, peer=557, peeraddr=31.211.105.232:37070 (full-relay)
2023-03-17T11:26:07Z New outbound peer connected: version: 70016, blocks=2061157, peer=558, peeraddr=178.217.159.221:37070 (full-relay)
2023-03-17T11:27:05Z New outbound peer connected: version: 70016, blocks=2133670, peer=559, peeraddr=209.55.111.169:37070 (full-relay)
2023-03-17T11:30:42Z New outbound peer connected: version: 70015, blocks=2140016, peer=560, peeraddr=77.173.8.2:37070 (full-relay)
2023-03-17T11:35:19Z Disconnecting outbound peer 553 for old chain, best known block = <none>
2023-03-17T11:37:32Z New outbound peer connected: version: 70016, blocks=2061157, peer=563, peeraddr=178.217.159.221:37070 (full-relay)
2023-03-17T11:40:10Z New outbound peer connected: version: 70016, blocks=2126978, peer=564, peeraddr=185.183.197.82:37070 (full-relay)
2023-03-17T11:41:16Z New outbound peer connected: version: 70015, blocks=2140035, peer=565, peeraddr=77.173.8.2:37070 (full-relay)
2023-03-17T11:45:28Z Disconnecting outbound peer 557 for old chain, best known block = <none>
2023-03-17T11:46:50Z New outbound peer connected: version: 70016, blocks=1450545, peer=566, peeraddr=31.211.105.232:37070 (full-relay)
2023-03-17T11:49:07Z Disconnecting outbound peer 559 for old chain, best known block = <none>
2023-03-17T11:49:23Z New outbound peer connected: version: 70016, blocks=2061157, peer=569, peeraddr=178.217.159.221:37070 (full-relay)
2023-03-17T11:50:15Z Potential stale tip detected, will try using extra outbound peer (last tip update: 328 seconds ago)
2023-03-17T11:53:14Z New outbound peer connected: version: 70016, blocks=2126978, peer=571, peeraddr=185.183.197.82:37070 (full-relay)
2023-03-17T11:58:52Z New outbound peer connected: version: 70016, blocks=2133670, peer=578, peeraddr=209.55.111.169:37070 (full-relay)
2023-03-17T11:59:42Z New outbound peer connected: version: 70015, blocks=2140050, peer=579, peeraddr=77.173.8.2:37070 (full-relay)
2023-03-17T12:05:16Z New outbound peer connected: version: 70016, blocks=2126978, peer=580, peeraddr=185.183.197.82:37070 (full-relay)
2023-03-17T12:06:41Z New outbound peer connected: version: 70016, blocks=2061157, peer=585, peeraddr=178.217.159.221:37070 (full-relay)
2023-03-17T12:08:52Z Disconnecting outbound peer 566 for old chain, best known block = <none>
2023-03-17T12:09:52Z New outbound peer connected: version: 70016, blocks=1452393, peer=587, peeraddr=31.211.105.232:37070 (full-relay)
2023-03-17T12:10:37Z New outbound peer connected: version: 70015, blocks=2140067, peer=588, peeraddr=77.173.8.2:37070 (full-relay)
2023-03-17T12:11:41Z Function `GetAccountProfiles` failed with execute timeout
2023-03-17T12:11:41Z ErrorLogCallback: 9; Message: statement aborts at 634: [
2023-03-17T12:12:41Z New outbound peer connected: version: 70016, blocks=2140591, peer=589, peeraddr=65.21.57.14:37070 (full-relay)
2023-03-17T12:16:51Z New outbound peer connected: version: 70016, blocks=2126978, peer=592, peeraddr=185.183.197.82:37070 (full-relay)
2023-03-17T12:17:50Z New outbound peer connected: version: 70016, blocks=2140614, peer=593, peeraddr=89.179.78.57:37070 (full-relay)
2023-03-17T12:20:54Z Disconnecting outbound peer 578 for old chain, best known block = <none>
2023-03-17T12:23:12Z New outbound peer connected: version: 70015, blocks=2140074, peer=597, peeraddr=77.173.8.2:37070 (full-relay)
2023-03-17T12:25:25Z New outbound peer connected: version: 70016, blocks=2061157, peer=598, peeraddr=178.217.159.221:37070 (full-relay)
2023-03-17T12:26:49Z New outbound peer connected: version: 70016, blocks=2133670, peer=601, peeraddr=209.55.111.169:37070 (full-relay)
2023-03-17T12:28:32Z New outbound peer connected: version: 70016, blocks=2126978, peer=603, peeraddr=185.183.197.82:37070 (full-relay)
2023-03-17T12:31:47Z Potential stale tip detected, will try using extra outbound peer (last tip update: 207 seconds ago)
2023-03-17T12:31:54Z Disconnecting outbound peer 587 for old chain, best known block = <none>
2023-03-17T12:33:42Z New outbound peer connected: version: 70015, blocks=2140086, peer=609, peeraddr=77.173.8.2:37070 (full-relay)
2023-03-17T12:34:43Z Disconnecting outbound peer 589 for old chain, best known block = <none>
2023-03-17T12:35:10Z New outbound peer connected: version: 70016, blocks=1454728, peer=610, peeraddr=31.211.105.232:37070 (full-relay)
2023-03-17T12:35:51Z New outbound peer connected: version: 70016, blocks=2061157, peer=611, peeraddr=178.217.159.221:37070 (full-relay)
2023-03-17T12:44:02Z New outbound peer connected: version: 70015, blocks=2140097, peer=615, peeraddr=77.173.8.2:37070 (full-relay)
In banner.log there were following lines regarding old nodes (time in UTC+3):
2023-03-16 13:25:52 === Old node version banned (0.20.29):
2023-03-16 13:25:52 178.209.98.206:12362 /Satoshi:0.20.29/ 2019352 -1 -1 328 7711 2023-03-16 13:25:02 inbound
2023-03-17 18:05:48 === Old node version banned (0.20.29):
2023-03-17 18:05:48 77.173.8.2:37070 /Satoshi:0.20.29/ 2140234 2077186 -1 165503 35575 2023-03-17 18:05:46 outbound-full-relay
What is the date of the first record in debug.log?
Near about 2023-03-16T21:50:00Z. Can't attach debug.log as a file.
Seems that my node was forked by 77.173.8.2 peer with 0.20.29 version.
The main catch is that the knot can fork because of one other node, but for no good reason it cannot stay in this fork for a long time, ignoring the main chain
Found connected node list dumped about 2023-03-16T23:06:35Z:
$ pocketcoin-cli getpeerinfo | jq -r '.[] | [.addr, .subver, .pingtime, .startingheight, .synced_headers, .synced_blocks, .bytesrecv, .bytessent, (.conntime|todate), .timeoffset, .connection_type] | @tsv' | column -t
95.31.45.162:37070 /Satoshi:0.21.1/ 0.012638 2139830 2139929 2139929 445910 9142 2023-03-16T21:58:53Z 0 block-relay-only
91.206.15.176:37070 /Satoshi:0.21.1/ 0.010894 2139830 2139931 2139931 952776 17250 2023-03-16T21:58:54Z -1 block-relay-only
73.96.245.93:37070 /Satoshi:0.21.1/ 0.201046 2139830 2139929 2139929 691146 251659 2023-03-16T21:59:16Z 1 outbound-full-relay
67.140.183.89:37070 /Satoshi:0.21.2/ 0.129884 2139831 2139929 2139929 498847 94594 2023-03-16T21:59:24Z 1 outbound-full-relay
135.181.196.243:37070 /Satoshi:0.21.2/ 0.017554 2139831 2139931 2139931 551241 419525 2023-03-16T21:59:34Z 2 outbound-full-relay
94.190.60.151:37070 /Satoshi:0.21.2/ 0.038438 2139831 2139931 2139931 312999 107954 2023-03-16T21:59:35Z 1 outbound-full-relay
162.246.52.155:37070 /Satoshi:0.21.1/ 0.132026 2139831 2139931 2139931 571476 96049 2023-03-16T21:59:35Z 1 outbound-full-relay
85.236.190.129:37070 /Satoshi:0.21.1/ 0.022291 2139831 2139931 2139931 1216752 95616 2023-03-16T21:59:54Z -1 outbound-full-relay
68.118.120.152:37070 /Satoshi:0.21.1/ 0.161434 2139832 2139929 2139929 161682 107349 2023-03-16T22:00:06Z 2 outbound-full-relay
185.9.187.123:54132 /Satoshi:0.21.2/ 0.018141 2139871 2139930 2139930 462779 90495 2023-03-16T22:30:16Z 2 inbound
37.48.87.107:37070 /Satoshi:0.21.2/ 0.042353 2139903 2139931 2139931 57315 20569 2023-03-16T23:06:35Z 1 outbound-full-relay
77.173.8.2 was not connected and therefore not banned by banner script! It was connected and banned only at 2023-03-17 18:05. I thought that one of 0.21 connected node was forked and then poisoned my node.
I am inclined to the fact that there are several nodes in the fork in the network. I have no idea how to return all nodes to the main chain - this situation is normal and should be resolved over time
In version 0.22, an optional disconnectold
argument has been added to disable old nodes or nodes in the fork - this should solve some of the problems with outdated nodes and the potential forks they cause.
Describe the bug Unrecoverable fork issued at 2140569. Current checkpoint (from 2023-03-18) provides forked chain at height 2140978 (best block: a34b9c3b175e89f394971041aeab914ab759a8437de15cf0230efaac42036cd5).
To Reproduce
Expected behavior Rollback to 2140568 block then validating and connecting 2140569 block to chain with valid hash (aeb0a0458361adbe49e7cca4330c9ce5128f52288947f13506984189678ba592).
Host and node version: