search

storj-archived / sips

Storj Improvement Proposals.
GNU General Public License v3.0
14 stars 11 forks source link

Download from specific positions of a file across multiple farmers -> improves file transfer speed #33

Open MeijeSibbel opened 7 years ago

MeijeSibbel commented 7 years ago

This is something that in combination with node benchmarking (selection based on for example bandwidth) would reduce the transfer time and increase transfer speed. Bittorrent does this well.

Reference: https://github.com/Storj/core/issues/705

braydonf commented 7 years ago

This should improve the performance towards the end of a file transfer where it's typical to see a decrease in speed when the concurrency decreases due to there only being a few shards left.

RGBKey commented 7 years ago

Am I incorrect in stating that streaming files from the Storj network would greatly benefit from this?

braydonf commented 7 years ago

Yes it could allow for better performance and bursts an specific positions. The issue with streaming though is that there can be wildly inconsistent behavior. If the first shard happens to be the shard than has gone missing and will need to be recovered, with reed solomon, it will be necessary to download enough data to be able to recover that shard, which would be a noticeable delay. I believe it's possible to do partial recovery without downloading the entire file, however it will still be more data than the size of the first shard.