Download Code Path Refactor

[iglesiasbrandon]

Summary:

Refactor the code that performs downloads inside libuplink to enable a collection of new features and improve upon lessons we've learned from the existing code.

Pain Points:

• eliminate stalls between segment downloads
• reduce the amount of startup roundtrips
• have dynamic long tail management
• be able to support more concurrency with downloading (largely to support pipelining with stream output, but it would also be useful to support much greater parallelism for seek-able writers (nice to have))
• enable us to do an uplink-lite (separating encryption from RS, so we can do uplink lite in the browser)
• minimize the buffer allocations, decrease the GC pressure
• optimize TTFB (currently full 32k buffer is downloaded and recovered before first byte is server)
• support different node selection scheme for downloads (like open 49 connections and use only the 29, where the connection open was the fastest)

Intended Outcome:

Fix all of the pain points!

Milestone:

TBD

[shaupt131]

Please see https://github.com/storj/storj/issues/5638 for findings that could help to support this work.

[iglesiasbrandon]

closing this roadmap item because its going to be completed in three separate independent parts

• 79

• 80

• 82