Discussion: Full streaming in Hyperdrive.

[martinheidegger]

Using Hyperdrive: when creating a write-stream in DAT, the logic will create a new version of a file for every chunk added. This allows to implement a distributed read-stream but following questions occur:

• How much data is going to be written to the stream? (Some streams - such as uploading a big zip file - can predict how much data will be part of the stream)
• Is it worth to keep the versions between the start of the stream and the end of the stream? (does metadata need to be sent for every block?)
• When is the write stream finished?
• What happens when a write stream is corrupted?

Currently the reading of streams is implemented by looking at the stat, and as soon as all the data for the stream arrived the stream is finished hyperdrive/index.js#L510

While the writing of streams appends a lot of data to the content-log hyperdrive/index.js#L578, it adds only one statement to the tree after finish hyperdrive/index.js#L598.

This means (to my understanding) that currently hyperdrive only starts a read-stream on a peer after a write-stream on the creators machine has entirely finished.

My straightforward idea to fix this would be that upon creation of a write-stream, hyperdrive could add a put message immediately to it: stating the streams final size if know or 0; adding a "open"-flag to the Stats. Upon finish there would be another tree.put with the final stats that don't contain the "open"-flag.

[Frando]

I coded something up recently: https://github.com/jimpick/hyperdrive/pull/1 The PR adds an option to hyperdrive to write the stat after each flush. It allowed me to view a video on a peer while still importing it on the source.