search

owncloud / ocis

:atom_symbol: ownCloud Infinite Scale Stack
https://doc.owncloud.com/ocis/next/
Apache License 2.0
1.36k stars 179 forks source link

design `upload discovery` tus extension #1298

Open butonic opened 4 years ago

butonic commented 4 years ago

To save clients from saving the upload id they would like to be able to discover existing upload ids using the path and a checksum.

This could be designed as a tus extension.

The client will send a post with the metadata and the server should be able to return either a 100-continue or a redirect the existing upload location.

cc @PVince81 @TheOneRing

michaelstingl commented 4 years ago

@felix-schwarz just FYI, no ETA yet

felix-schwarz commented 4 years ago

Nice! That would be super useful to resume a create-with-upload request that was interrupted.

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 10 days if no further activity occurs. Thank you for your contributions.

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 10 days if no further activity occurs. Thank you for your contributions.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 10 days if no further activity occurs. Thank you for your contributions.

fschade commented 3 months ago

Please open again if the ticket is still relevant

micbar commented 3 months ago

@TheOneRing @butonic which end user value does that bring?

TheOneRing commented 3 months ago

By default tus doesn't not use chunks, so if you upload a 100GiB, with creation-with-upload, and the upload fails, we never received an upload location to continue and have to start all over again.

We currently work around this issue by using ca 100MiB chunks, but ideally we could roll an id for creation-with-upload.

michaelstingl commented 3 months ago

We currently work around this issue by using ca 100MiB chunks

Backend now announces 10 MiB:

https://github.com/owncloud/ocis/blob/b892a9ab286f85a8db7b61164ff234131443ac50/docs/apis/http/tus_upload.md?plain=1#L20-L44

Related:

michaelstingl commented 3 months ago

but ideally we could roll an id for creation-with-upload.

This is the Upload Tag extension in tus:1.1+

For this approach, clients roll an id (Upload Tag), but need to remember it to query information about the incomplete upload later.

This issue is about an approach, where clients wouldn't need to persist the id (Upload Tag), but instead use the path and a checksum.

To save clients from saving the upload id they would like to be able to discover existing upload ids using the path and a checksum.

micbar commented 3 months ago

@TheOneRing @michaelstingl Let me summarize:

Like @michaelstingl wrote, we have the default of 10MB for the chunk size these days. I am a bit hesitant to add "too much extra stuff" in the light of TUS becoming a standard protocol for vendors like apple.

https://tus.io/blog/2023/08/09/resumable-uploads-ietf

Parts of the IETF proposal

1718088693313

create and creation-with-upload will be part of that. Bottom line from my POV is, that we should not go beyond that.

TheOneRing commented 3 months ago

@TheOneRing @michaelstingl Let me summarize:

* The clients want to be able to resume uploads without "remembering" the upload ID

The client wants to be able to suggest the upload ID, if we only get the upload ID as a response to the first request, we can't resume anything if that request failed.

But I have no strong opinion on this, I just didn't want to see it closed because "Old"

* The clients want to be able to recover from a failed `creation-with-upload` request.

Like @michaelstingl wrote, we have the default of 10MB for the chunk size these days. I am a bit hesitant to add "too much extra stuff" in the light of TUS becoming a standard protocol for vendors like apple.

https://tus.io/blog/2023/08/09/resumable-uploads-ietf

Parts of the IETF proposal

1718088693313

create and creation-with-upload will be part of that. Bottom line from my POV is, that we should not go beyond that.

felix-schwarz commented 3 months ago

@micbar @TheOneRing IIRC this issue originated from an early discussion - which eventually culminated in me proposing Upload-Token to the tus team and working with them to add it to the spec.

While - technically - Upload-Token is a different thing I'm not sure an issue of its own exists for it here.

With regards to the iOS client, supporting Upload-Token can

In short: it would elevate the possible app experience.

That vendors like Apple have started adding basic tus support to their OS is a good point, but it will likely take a couple more releases (years) until we get full-fledged support - and then a few more years until that OS version is old enough to become the minimum required version for running the app. So there's still a long time to go until the iOS app can depend on OS support for tus.