search

paritytech / polkadot-sdk

The Parity Polkadot Blockchain SDK
https://polkadot.network/
1.78k stars 640 forks source link

Node not syncing anymore after importing block in reverse order #535

Open crystalin opened 3 years ago

crystalin commented 3 years ago

After doing a purge of the network, one of our bootnode got stuck at a given block height. Even after restarting it gets stuck. Something weird from the log is the fact it imported blocks in the reversed order:

2021-09-03 18:13:38.255  INFO tokio-runtime-worker substrate: [🌗] ✨ Imported #456811 (0x5a91…1151)
2021-09-03 18:13:38.567  INFO tokio-runtime-worker substrate: [🌗] ✨ Imported #456802 (0x1966…c26c)

Full logs:

2021-09-03 18:13:16.973  INFO tokio-runtime-worker substrate_prometheus_endpoint::known_os: [Relaychain] 〽️ Prometheus server started at 0.0.0.0:9615
2021-09-03 18:13:17.019  INFO tokio-runtime-worker substrate_prometheus_endpoint::known_os: [🌗] 〽️ Prometheus server started at 0.0.0.0:9616
2021-09-03 18:13:21.976  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (47 peers), best: #9067416 (0x86ec…f4b6), finalized #9067414 (0x35e1…081f), ⬇ 339.3kiB/s ⬆ 623.9kiB/s
2021-09-03 18:13:22.024  INFO tokio-runtime-worker substrate: [🌗] 💤 Idle (18 peers), best: #456804 (0x8a9c…d7aa), finalized #0 (0x401a…474b), ⬇ 70.1kiB/s ⬆ 123.1kiB/s
2021-09-03 18:13:26.977  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (47 peers), best: #9067416 (0x86ec…f4b6), finalized #9067414 (0x35e1…081f), ⬇ 681.2kiB/s ⬆ 508.9kiB/s
2021-09-03 18:13:27.024  INFO tokio-runtime-worker substrate: [🌗] 💤 Idle (20 peers), best: #456804 (0x8a9c…d7aa), finalized #0 (0x401a…474b), ⬇ 230.5kiB/s ⬆ 116.7kiB/s
2021-09-03 18:13:31.977  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (58 peers), best: #9067416 (0x86ec…f4b6), finalized #9067415 (0x7dd3…0fd7), ⬇ 658.3kiB/s ⬆ 936.9kiB/s
2021-09-03 18:13:32.025  INFO tokio-runtime-worker substrate: [🌗] 💤 Idle (31 peers), best: #456804 (0x8a9c…d7aa), finalized #0 (0x401a…474b), ⬇ 130.2kiB/s ⬆ 225.9kiB/s
2021-09-03 18:13:33.815  INFO tokio-runtime-worker substrate: [🌗] ✨ Imported #456806 (0x8c13…a6fb)
2021-09-03 18:13:36.825  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #9067417 (0xe7dd…69b5)
2021-09-03 18:13:36.987  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (59 peers), best: #9067417 (0xe7dd…69b5), finalized #9067416 (0x86ec…f4b6), ⬇ 904.6kiB/s ⬆ 1.8MiB/s
2021-09-03 18:13:37.028  INFO tokio-runtime-worker substrate: [🌗] 💤 Idle (43 peers), best: #456810 (0x0d47…385b), finalized #0 (0x401a…474b), ⬇ 75.0kiB/s ⬆ 502.3kiB/s
2021-09-03 18:13:37.465  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #9067418 (0xce91…4ecc)
2021-09-03 18:13:37.594  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #9067419 (0xf99c…10cd)
2021-09-03 18:13:37.745  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #9067420 (0x11b5…fd17)
2021-09-03 18:13:37.807  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #9067420 (0x8e33…4870)
2021-09-03 18:13:37.879  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #9067421 (0xf4a4…8aa6)
2021-09-03 18:13:37.940  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #9067422 (0x2543…fb29)
2021-09-03 18:13:38.255  INFO tokio-runtime-worker substrate: [🌗] ✨ Imported #456811 (0x5a91…1151)
2021-09-03 18:13:38.567  INFO tokio-runtime-worker substrate: [🌗] ✨ Imported #456802 (0x1966…c26c)
2021-09-03 18:13:41.987  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (62 peers), best: #9067422 (0x2543…fb29), finalized #9067418 (0xce91…4ecc), ⬇ 1.2MiB/s ⬆ 1.9MiB/s
2021-09-03 18:13:42.028  INFO tokio-runtime-worker substrate: [🌗] 💤 Idle (46 peers), best: #456811 (0x5a91…1151), finalized #0 (0x401a…474b), ⬇ 59.8kiB/s ⬆ 279.3kiB/s
2021-09-03 18:13:42.596  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #9067423 (0x522d…3e84)
2021-09-03 18:13:46.987  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (66 peers), best: #9067423 (0x522d…3e84), finalized #9067418 (0xce91…4ecc), ⬇ 1.4MiB/s ⬆ 464.1kiB/s
2021-09-03 18:13:47.030  INFO tokio-runtime-worker substrate: [🌗] 💤 Idle (46 peers), best: #456811 (0x5a91…1151), finalized #0 (0x401a…474b), ⬇ 28.1kiB/s ⬆ 18.5kiB/s
2021-09-03 18:13:48.307  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #9067424 (0xc403…6fb5)
2021-09-03 18:13:51.988  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (67 peers), best: #9067424 (0xc403…6fb5), finalized #9067418 (0xce91…4ecc), ⬇ 1.8MiB/s ⬆ 908.8kiB/s
2021-09-03 18:13:52.030  INFO tokio-runtime-worker substrate: [🌗] 💤 Idle (48 peers), best: #456811 (0x5a91…1151), finalized #0 (0x401a…474b), ⬇ 49.3kiB/s ⬆ 63.8kiB/s
2021-09-03 18:13:54.550  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #9067425 (0x7a18…dc12)
2021-09-03 18:13:56.988  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (68 peers), best: #9067425 (0x7a18…dc12), finalized #9067418 (0xce91…4ecc), ⬇ 1.5MiB/s ⬆ 507.4kiB/s
2021-09-03 18:13:57.030  INFO tokio-runtime-worker substrate: [🌗] 💤 Idle (51 peers), best: #456811 (0x5a91…1151), finalized #0 (0x401a…474b), ⬇ 15.7kiB/s ⬆ 146.1kiB/s
2021-09-03 18:14:00.739  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #9067426 (0x70fc…9dc0)
2021-09-03 18:14:01.988  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (68 peers), best: #9067426 (0x70fc…9dc0), finalized #9067420 (0x11b5…fd17), ⬇ 1.6MiB/s ⬆ 614.9kiB/s
2021-09-03 18:14:02.030  INFO tokio-runtime-worker substrate: [🌗] 💤 Idle (50 peers), best: #456811 (0x5a91…1151), finalized #0 (0x401a…474b), ⬇ 20.3kiB/s ⬆ 26.8kiB/s
2021-09-03 18:14:06.358  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #9067427 (0x880d…2a4c)
2021-09-03 18:14:06.640  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #9067427 (0x3d89…0244)
2021-09-03 18:14:06.988  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (69 peers), best: #9067427 (0x880d…2a4c), finalized #9067421 (0xf4a4…8aa6), ⬇ 1.5MiB/s ⬆ 786.8kiB/s
2021-09-03 18:14:07.031  INFO tokio-runtime-worker substrate: [🌗] 💤 Idle (50 peers), best: #456811 (0x5a91…1151), finalized #0 (0x401a…474b), ⬇ 17.1kiB/s ⬆ 7.7kiB/s
2021-09-03 18:14:11.989  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (70 peers), best: #9067427 (0x880d…2a4c), finalized #9067422 (0x2543…fb29), ⬇ 1.5MiB/s ⬆ 1.4MiB/s
2021-09-03 18:14:12.031  INFO tokio-runtime-worker substrate: [🌗] 💤 Idle (50 peers), best: #456811 (0x5a91…1151), finalized #0 (0x401a…474b), ⬇ 9.3kiB/s ⬆ 7.0kiB/s
2021-09-03 18:14:12.401  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #9067428 (0x1e42…e3cb)
bkchr commented 2 years ago

I mean the log line is very confusing and not sure how that even works. However, importing in reverse order is impossible :D

Next time enable sync=trace so we know why it stopped.