search

storj / uplink

Storj network Go library
MIT License
115 stars 18 forks source link

Eliminate unnecessary buffer allocations #122

Closed iglesiasbrandon closed 1 year ago

iglesiasbrandon commented 1 year ago

The goal here is refactor upload (and later download) to reduce usage of buffer allocations.

In the current model we use io.Copy and buffer based approaches to copy data between different abstractions.

The goal here is to rethink the abstraction to do as few buffer copy as possible.

As a side-effect: with better abstraction we can separate the different functions easier: it can be possible to do an encryption locally, but calculate EC and connect to storagenodes from a lightweight gateway service.

A possible approach is evaluated in the https://github.com/storj/uplink/tree/uploadng branch.

This is not a fully working solution, just a good skeleton which contains all the important part. Initial performance tests (based on unit tests) shows that it can be significant faster.

However the full implementation requires either further improvements on the abstraction or simplification in the protocol change, as not all the encoding information is available in advance...

iglesiasbrandon commented 1 year ago

closing this issue because its actually just the acceptance criteria for this effort