Download Code Path Refactor Blueprint

storj / uplink

Storj network Go library

MIT License

115 stars 18 forks source link

Download Code Path Refactor Blueprint #144

Closed iglesiasbrandon closed 11 months ago

iglesiasbrandon commented 1 year ago

Description

This issue aims to create a blueprint for the download code path refactor. This refactor would eliminate the pause between segment downloads.

Acceptance Criteria:

A blueprint or tickets are created to tackle this effort.

iglesiasbrandon commented 1 year ago

@zeebo @jtolio We created this ticket, but I'm not entirely sure of the scope of this effort. Can you shed some light on this for me?

what are the current pain points?
what would this refactor solve or help with?

jtolio commented 1 year ago

largely the same things as the upload codepath refactor. we would like to:

eliminate stalls between segment downloads
reduce the amount of startup roundtrips
have dynamic long tail management
be able to support more concurrency with downloading (largely to support pipelining with stream output, but it would also be useful to support much greater parallelism for seekable writers (nice to have))
lastly, to enable us to do an uplink-lite (separating encryption from RS, so we can do uplink lite in the browser)

elek commented 1 year ago

minimize the buffer allocations, decrease the GC pressure
optimize TTFB (currently full 32k buffer is downloaded and recovered before first byte is server)

elek commented 1 year ago

support different node selection scheme for downloads (like open 49 connections and use only the 29, where the connection open was the fastest)

iglesiasbrandon commented 1 year ago

We are going to close this ticket because a formal blueprint is not needed. instead what we will do is write a short description of how this is intended to work on the roadmap item itself. @zeebo

https://github.com/storj/roadmap/issues/71

shaupt131 commented 1 year ago

Looks like @elek captured our thoughts on the roadmap ticket. Closing this issue.

iglesiasbrandon commented 1 year ago

@shaupt131 we are reopening this issue we need to write a blueprint for the download code path refactor

iglesiasbrandon commented 11 months ago

we created three separate items to complete the overall download code path refactor. https://github.com/storj/roadmap/issues/79 https://github.com/storj/roadmap/issues/80 https://github.com/storj/roadmap/issues/82