ethereum / utp

uTorrent transport protocol
MIT License
28 stars 13 forks source link

Fix large transfers #132

Closed carver closed 3 months ago

carver commented 3 months ago

@njgheorghita this is the fix for the 1MiB transfer failures you were seeing in the "gossip on delete" prototype.

Problem: transfers with single writes over ~1MB were hanging, then failing by timeout.

Basic approach:

Unfortunately, there does appear to be some kind of performance hit when the transfer is much larger than the buffer. See the speed drop off massively in these 5 different tests:

2024-08-02T01:45:20.923482Z  INFO socket: finished udp load test with 1 transfer of 1 MB, in 14.0ms, at a rate of 597.4 Mbps
2024-08-02T01:46:02.074239Z  INFO socket: finished udp load test with 1 transfer of 2 MB, in 62.9ms, at a rate of 266.6 Mbps
2024-08-02T01:55:20.269157Z  INFO socket: finished udp load test with 1 transfer of 4 MB, in 582.3ms, at a rate of 57.6 Mbps
2024-08-02T02:44:33.315410Z  INFO socket: finished udp load test with 1 transfer of 34 MB, in 6.5s, at a rate of 41.3 Mbps
2024-08-02T02:45:45.387391Z  INFO socket: finished udp load test with 1 transfer of 52 MB, in 22.6s, at a rate of 18.5 Mbps

I haven't had a chance to hunt down the cause. At the moment, I think I will just move on. We really need to get this bugfix out now, and I think there are more urgent things that need to be resolved before Devcon.

Note that concurrent transfers can still saturate UDP, at ~1Gbps:

2024-08-02T04:53:51.354249Z  INFO socket: finished high concurrency load test of 1000 simultaneous transfers, in 7.72s, at a rate of 1036 Mbps
KolbyML commented 3 months ago

Unfortunately, there does appear to be some kind of performance hit when the transfer is much larger than the buffer. See the speed drop off massively in these 5 different tests:

this is very interesting