madecoste / swarming

Automatically exported from code.google.com/p/swarming
Apache License 2.0
0 stars 1 forks source link

Isolate download saturate network I/O in case of mixture of large number of small files (>10k) and small number of large files #146

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago
Goal:
Saturate network I/O when downloading.

We didn't tune isolateserver much about large files download performance, so 
there is likely low hanging fruits to be determined by profiling.

Proposed implementations:
These are all to be done entirely client side:
- When large files are downloaded, limit the number of parallel streams to a 
few. The idea is that we don't want to starve large downloads too much so their 
TCP connection get throttled, creating laggards.
- When small files are downloaded, increases connection parallelism to reduce 
the negative effect of HTTP latency overhead (even when taking in account TCP 
connection reuse) because of the latency of HTTP 302's that isolate server is 
just too happy to reply with.
- When only large files remain, start chunked download.

Original issue reported on code.google.com by maruel@chromium.org on 15 Aug 2014 at 5:31