paritytech / polkadot-sdk

The Parity Polkadot Blockchain SDK
https://polkadot.network/
1.85k stars 667 forks source link

`fatxpool`: networking::transaction subsystem `clogged` issue with many transactions #5486

Open michalkucharczyk opened 1 month ago

michalkucharczyk commented 1 month ago

When executing spamming tests, there is significant number of the following debugs:

[Parachain] Libp2p <= Starting random Kademlia request for PeerId("1AXGT2YqJCUXDZpYijnFPMnSuahK1tQM5gUHTNv9j2jQ1L")
[Parachain] Libp2p => Disconnected(PeerId("12D3KooWLQQU12Kz69ai5QyHvMi4oRm4xsgAHKpZ5J1XcZvNBWaY") via ConnectionId(135), Some(Handler(Left(Left(Left(Right(SyncNotificationsClogged)))))))
[Parachain] Libp2p => IncomingConnectionError(/ip4/192.168.1.200/tcp/33191/ws,/ip4/192.168.1.200/tcp/56640/ws via ConnectionId(175)): Listen error: Failed to negotiate transport protocol(s)
[Parachain] Libp2p => IncomingConnectionError(/ip4/127.0.0.1/tcp/33191/ws,/ip4/127.0.0.1/tcp/33446/ws via ConnectionId(174)): Listen error: Failed to negotiate transport protocol(s)
[Parachain] Libp2p => IncomingConnectionError(/ip4/172.17.0.1/tcp/33191/ws,/ip4/192.168.1.200/tcp/34684/ws via ConnectionId(176)): Listen error: Failed to negotiate transport protocol(s)
[Parachain] Libp2p => Connected(PeerId("12D3KooWLQQU12Kz69ai5QyHvMi4oRm4xsgAHKpZ5J1XcZvNBWaY"))

This is probably caused becuase the number of transactions is quite high (8k+), while the value of some internal buffer is quite small comparing to that: SYNC_NOTIFICATIONS_BUFFER_SIZE. Increasing this constant helps, not sure what are the consequences. This issue needs further investigation, especially for networks which require high transaction throuput (e.g. 4k+ transactions per block, with all blocks filled for long period of time).

Test that failed:

ttxt  tx --unwatched --block-monitor --send-threshold=5000  --chain=eth --ws=ws://127.0.0.1:9933 from-single-account --account alice --count 100000
michalkucharczyk commented 1 week ago

useful debug: https://github.com/paritytech/polkadot-sdk/blob/5a5fb09577de4b67bc28302ac72a2f35e734ae08/substrate/client/network/transactions/src/lib.rs#L235-L243