paritytech / polkadot-sdk

The Parity Polkadot Blockchain SDK
https://polkadot.network/
1.77k stars 633 forks source link

network/litep2p: Slower sync time compared to libp2p #4986

Open lexnv opened 2 months ago

lexnv commented 2 months ago
Stage litep2p libp2p
Warp Sync 9m 52s 8m 27s
State Sync 6m 32s 5m 30s
Total Sync Time 16m 24s 13m 57s

litep2p

2024-07-05 14:06:42.414  INFO main sub-libp2p: Running litep2p network backend

2024-07-05 14:06:47.449  INFO tokio-runtime-worker substrate: ⏩ Warping, Downloading finality proofs, 0.00 Mib (4 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 845.8kiB/s ⬆ 5.8kiB/s
2024-07-05 14:06:52.449  INFO tokio-runtime-worker substrate: ⏩ Warping, Downloading finality proofs, 7.98 Mib (39 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 2.4MiB/s ⬆ 17.6kiB/s
2024-07-05 14:07:31.166  INFO tokio-runtime-worker substrate: ⏩ Warping, Downloading finality proofs, 95.53 Mib (50 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 3.7MiB/s ⬆ 6.8kiB/s

2024-07-05 14:16:34.688  INFO tokio-runtime-worker sync: Warp sync is complete, continuing with state sync.

2024-07-05 14:16:35.792  INFO tokio-runtime-worker substrate: ⚙️  State sync, Downloading state, 0%, 0.00 Mib (52 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 4.0MiB/s ⬆ 7.1kiB/s

2024-07-05 14:22:25.878  INFO tokio-runtime-worker substrate: ⚙️  State sync, Downloading state, 83%, 384.47 Mib (52 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 4.8MiB/s ⬆ 10.9kiB/s
2024-07-05 14:22:30.878  INFO tokio-runtime-worker substrate: ⚙️  State sync, Downloading state, 83%, 389.20 Mib (52 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 2.8MiB/s ⬆ 15.0kiB/s
2024-07-05 14:22:35.879  INFO tokio-runtime-worker substrate: ⚙️  State sync, Importing state, 84%, 393.82 Mib (52 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 3.0MiB/s ⬆ 12.9kiB/s
2024-07-05 14:22:40.879  INFO tokio-runtime-worker substrate: ⚙️  State sync, Importing state, 84%, 393.82 Mib (52 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 2.0MiB/s ⬆ 23.1kiB/s
2024-07-05 14:22:45.879  INFO tokio-runtime-worker substrate: ⚙️  State sync, Importing state, 84%, 393.82 Mib (52 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 2.1MiB/s ⬆ 37.9kiB/s
2024-07-05 14:22:50.879  INFO tokio-runtime-worker substrate: ⚙️  State sync, Importing state, 84%, 393.82 Mib (52 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 2.0MiB/s ⬆ 22.1kiB/s
2024-07-05 14:22:55.880  INFO tokio-runtime-worker substrate: ⚙️  State sync, Importing state, 84%, 393.82 Mib (52 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 2.1MiB/s ⬆ 22.2kiB/s
2024-07-05 14:23:00.880  INFO tokio-runtime-worker substrate: ⚙️  State sync, Importing state, 84%, 393.82 Mib (52 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 2.2MiB/s ⬆ 12.6kiB/s
2024-07-05 14:23:05.881  INFO tokio-runtime-worker substrate: ⚙️  State sync, Importing state, 84%, 393.82 Mib (52 peers), best: #23910728 (0x59fd…5894), finalized #23910728 (0x59fd…5894), ⬇ 2.0MiB/s ⬆ 9.6kiB/s
2024-07-05 14:23:06.679  INFO tokio-runtime-worker sync: State sync is complete, continuing with block sync.
2024-07-05 14:23:10.881  INFO tokio-runtime-worker substrate: ⏩ Block history, #2240 (49 peers), best: #23910793 (0x31fe…ba62), finalized #23910728 (0x59fd…5894), ⬇ 4.0MiB/s ⬆ 8.2kiB/s
2024-07-05 14:23:12.648  INFO tokio-runtime-worker substrate: 🏆 Imported #23910797 (0xaacd…51f4 → 0x542a…a63b)

Libp2p

2024-07-05 14:07:27.569  INFO main sub-libp2p: Running libp2p network backend

2024-07-05 14:07:33.544  INFO tokio-runtime-worker substrate: ⏩ Warping, Downloading finality proofs, 7.98 Mib (38 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 3.1MiB/s ⬆ 80.4kiB/s
2024-07-05 14:07:38.544  INFO tokio-runtime-worker substrate: ⏩ Warping, Downloading finality proofs, 23.98 Mib (42 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 5.3MiB/s ⬆ 8.4kiB/s
2024-07-05 14:07:49.954  INFO tokio-runtime-worker substrate: ⏩ Warping, Downloading finality proofs, 63.76 Mib (51 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 3.9MiB/s ⬆ 3.0kiB/s

2024-07-05 14:15:54.968  INFO tokio-runtime-worker sync: Warp sync is complete, continuing with state sync.

2024-07-05 14:15:58.835  INFO tokio-runtime-worker substrate: ⚙️  State sync, Downloading state, 9%, 6.09 Mib (51 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 4.1MiB/s ⬆ 5.0kiB/s
2024-07-05 14:16:03.835  INFO tokio-runtime-worker substrate: ⚙️  State sync, Downloading state, 14%, 8.41 Mib (51 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 1.9MiB/s ⬆ 0.8kiB/s

2024-07-05 14:20:33.850  INFO tokio-runtime-worker substrate: ⚙️  State sync, Downloading state, 80%, 358.72 Mib (51 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 3.4MiB/s ⬆ 4.9kiB/s
2024-07-05 14:20:38.887  INFO tokio-runtime-worker substrate: ⚙️  State sync, Downloading state, 83%, 368.67 Mib (51 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 2.9MiB/s ⬆ 0.5kiB/s
2024-07-05 14:20:43.887  INFO tokio-runtime-worker substrate: ⚙️  State sync, Downloading state, 83%, 377.42 Mib (51 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 4.1MiB/s ⬆ 1.3kiB/s
2024-07-05 14:20:48.888  INFO tokio-runtime-worker substrate: ⚙️  State sync, Downloading state, 83%, 386.87 Mib (51 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 3.8MiB/s ⬆ 0.9kiB/s
2024-07-05 14:20:53.888  INFO tokio-runtime-worker substrate: ⚙️  State sync, Importing state, 84%, 393.83 Mib (51 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 2.5MiB/s ⬆ 1.1kiB/s
2024-07-05 14:20:58.888  INFO tokio-runtime-worker substrate: ⚙️  State sync, Importing state, 84%, 393.83 Mib (51 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 1.5MiB/s ⬆ 1.0kiB/s
2024-07-05 14:21:03.889  INFO tokio-runtime-worker substrate: ⚙️  State sync, Importing state, 84%, 393.83 Mib (51 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 1.6MiB/s ⬆ 0.7kiB/s
2024-07-05 14:21:08.889  INFO tokio-runtime-worker substrate: ⚙️  State sync, Importing state, 84%, 393.83 Mib (51 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 1.6MiB/s ⬆ 1.9kiB/s
2024-07-05 14:21:13.889  INFO tokio-runtime-worker substrate: ⚙️  State sync, Importing state, 84%, 393.83 Mib (51 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 1.5MiB/s ⬆ 0.5kiB/s
2024-07-05 14:21:18.889  INFO tokio-runtime-worker substrate: ⚙️  State sync, Importing state, 84%, 393.83 Mib (51 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 1.5MiB/s ⬆ 0.7kiB/s
2024-07-05 14:21:23.890  INFO tokio-runtime-worker substrate: ⚙️  State sync, Importing state, 84%, 393.83 Mib (51 peers), best: #23910722 (0x4cae…26e0), finalized #23910722 (0x4cae…26e0), ⬇ 1.5MiB/s ⬆ 0.9kiB/s
2024-07-05 14:21:24.293  INFO tokio-runtime-worker sync: State sync is complete, continuing with block sync.

2024-07-05 14:21:38.891  INFO tokio-runtime-worker substrate: ⏩ Block history, #9408 (50 peers), best: #23910781 (0x3b7b…9381), finalized #23910778 (0x2c10…3546), ⬇ 6.8MiB/s ⬆ 189.1kiB/s
2024-07-05 14:21:42.386  INFO tokio-runtime-worker substrate: 🏆 Imported #23910782 (0x3b7b…9381 → 0x2c56…2b22)

Data points

litep2p warp sync: 14:06:42.414 - 14:16:34.688 = 9m 52s libp2p warp sync: 14:07:27.569 - 14:15:54.968 = 8m 27s

litep2p state sync: 14:16:34.688 - 14:23:06.679 = 6m 32s libp2p state sync: 14:15:54.968 - 14:21:24.293 = 5m 30s

lexnv commented 2 months ago

This may free up some litep2p resources:

Based on comment litep2p handles considerably more connections.

Might also be the cause of a lower networking bytes sent:

lexnv commented 1 month ago

After implementing connection limits in litep2p:

Stage litep2p libp2p
Warp Sync 6m 38s 18m 35s
State Sync 4m 10s 1m 58s
Total Sync Time 10m 48s 20m 33s

Based on origin/master at ce6938ae92b.

After implementing connection limits, we have effectively improved the sync time with around 50%, from 16m 24s to 10m 48s.

We are also seeing a regression for libp2p, related to:

In the current state of things, litep2p is twice is fast as libp2p. Even compared to the previous better libp2p state, we are still 30% faster in syncing.

It is clear we need more performance testing, both for:

(cc @alexggh @sandreim )

dmitry-markin commented 1 month ago

Why are the numbers for libp2p so different? Comparing the first message with this one, warp sync went 8m 27s -> 18m 35s, state sync 5m 30s -> 1m 58s. Are they for the different chains, or the measurements are just noisy? (The same might apply for litep2p.)

lexnv commented 1 month ago

It could be some noise for sure. I've synced to kusama both nodes, believe I've used the same params here

We could double check using more data points, and probably improve https://github.com/lexnv/sub-triage-logs/ to do this for us.