search

tus / tus-resumable-upload-protocol

Open Protocol for Resumable File Uploads
https://tus.io
MIT License
1.49k stars 102 forks source link

seekable uploads #109

Closed ronag closed 5 years ago

ronag commented 7 years ago

This is a rather large extension.

Consider the scenario where a client is trying to upload a quicktime MOV file and the server wants to be able to stream it directly. In order to do this the server will require the file trailer. This requires that the server can hint the client which parts of the file should be uploaded first.

Essentially the server needs to be able to create and dynamically update the Concat list where the client will upload the parts in the order hinted by the server, e.g.

[header][trailer][part1][part2]...

An alternative to this could be that the client uploads everything in order as usual, but uses PATCH chunks of e.g. 64 mb, then the server gets a range request for a part that is not yet uploaded, then it could set the upload-offset for the next PATCH to the start of the requested range-request. So upload-offset is no longer the end of the file but the seek point where the server wants to receive the next data.

Acconut commented 7 years ago

I believe that this can already be achieved in a different fashion using the Concatenation extension while yielding the same result. The process would be following:

The client creates one partial upload for the header, the trailer and the every other part. In order to help the server know what kind each upload is, you can supply custom metadata, e.g. type=header or type=trailer. Right before beginning to upload the actual file content, the client creates a new final upload which is created by concatenation the partial uploads in the correct order, e.g. header, part #1, ..., part #N, trailer. After this the client will first upload the header followed by the trailer and in the end each part.

This way the server will receive the header and trailer first to obtain the required information while still resulting in a file which is a valid MOV format.

What do you think?

cjhenck commented 7 years ago

Is the final upload GET-able even before all of the parts have been uploaded?

Makes sense in terms of process.

Acconut commented 7 years ago

Is the final upload GET-able even before all of the parts have been uploaded?

The specification does not define the GET operation so this is just my recommendation. I don't think that it would make sense to serve the content in this case as it will probably be invalid, e.g. if the server just has the header and trailer.

cjhenck commented 7 years ago

Makes sense given the undefined GET, but given the streaming use case that @ronag mentions I think as long as there is a header and a footer, that the middle of the file can be streamed without the rest of it as suggested by @ronag's comment:

This requires that the server can hint the client which parts of the file should be uploaded first.

That said, I feel like this is a very customized use case since each HEAD or PATCH would need to respond to let the client know which chunks are coming up.

Acconut commented 7 years ago

Makes sense given the undefined GET, but given the streaming use case that @ronag mentions I think as long as there is a header and a footer, that the middle of the file can be streamed without the rest of it as suggested by @ronag's comment:

If you stream the download, then that would be possible, yes.

That said, I feel like this is a very customized use case since each HEAD or PATCH would need to respond to let the client know which chunks are coming up.

I agree, this is a rather specialized use case which needs a dedicated upload handler.

Acconut commented 5 years ago

Closing this issue due to inactivity. Feel free to leave a comment if you want to continue the discussion :)