iterate-ch / cyberduck

Cyberduck is a libre FTP, SFTP, WebDAV, Amazon S3, Backblaze B2, Microsoft Azure & OneDrive and OpenStack Swift file transfer client for Mac and Windows.
https://cyberduck.io/
GNU General Public License v3.0
3.12k stars 284 forks source link

Segmented WebDAV upload #13652

Open artshade opened 1 year ago

artshade commented 1 year ago

Dear Developers,

Thank you very much for the awesome project.


Is your feature request related to a problem? Please describe.

For example, a WebDAV server has a maximum upload size limit set to 512 MB. An upload of 1 GB file will result in an error on the server side like the following or similar: client intended to send too large body. The upload fails.

Describe the solution you'd like

What if either of these:

Describe alternatives you've considered

Other protocols which support resume or segmented/buffer transfers, like SFTP/FTPS/FTP or BitTorrent.

Additional context

https://www.rfc-editor.org/rfc/rfc9110.html#section-14.4 (The "Content-Range" header field is sent in...)
https://www.rfc-editor.org/rfc/rfc9110.html#section-14.5 (Partial PUT...)
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Range (The Content-Range response HTTP header indicates where...)
https://stackoverflow.com/q/716680/5113030 (Difference between Content-Range and Range headers?...)


Best and kind regards

dkocher commented 1 year ago

Are you aware of a server implementation that supports partial PUT requests with Content-Range headers?

artshade commented 1 year ago

Are you aware of a server implementation that supports partial PUT requests with Content-Range headers?

I am sorry, but may you provide more insight? It seems the issue description includes a suggestion and references of the technology you mentioned. Perhaps more details will help!

AliveDevil commented 1 year ago

The HTTP Content-Range and Partial Put aren't supported on many Dav-servers, and there are conflicting implementations. Namely Apache mod_dav uses PUT with Content-Range header, and SabreDav (Nextcloud, Owncloud, and many others) use PATCH with X-Update-Range header - see rust webdav_handler^1.

artshade commented 1 year ago

The HTTP Content-Range and Partial Put aren't supported on many Dav-servers, and there are conflicting implementations. Namely Apache mod_dav uses PUT with Content-Range header, and SabreDav (Nextcloud, Owncloud, and many others) use PATCH with X-Update-Range header - see rust webdav_handler1.

Footnotes

  1. https://docs.rs/webdav-handler/0.2.0-alpha.1/webdav_handler/#implemented-standards

Thank you very much for the reference. It's known that it's completely up to a server, as also stated in the RFCs above. The question is should the client support at least either of these (e.g. Nextcloud on Nginx)?

P.S. Very nice reference design, @AliveDevil! Off-topic, but I am curious to ask how would you add a reference to your message which is not tied to any direct part of any sentence, but just an additional/contextual (without a number/index)?

Related:
https://docs.nextcloud.com/server/latest/developer_manual/client_apis/WebDAV/chunking.html (Nextcloud has a chunking API where you can upload smaller chunks...)
https://docs.nextcloud.com/server/latest/admin_manual/configuration_files/big_file_upload_configuration.html (Uploading big files > 512MB...)

AliveDevil commented 1 year ago

P.S. Very nice reference design, @AliveDevil! Off-topic, but I am curious to ask how would you add a reference to your message which is not tied to any direct part of any sentence, but just an additional/contextual (without a number/index)?

These are footnotes, as described in Markdown standard, and implemented by Github since September 2021 in comments^1.

Footnotes unfortunately don't work without direct reference, so you'd need [^1][^2][^3] at the end of your message, to include additional context.

ArcticLampyrid commented 6 months ago

The HTTP Content-Range and Partial Put aren't supported on many Dav-servers, and there are conflicting implementations. Namely Apache mod_dav uses PUT with Content-Range header, and SabreDav (Nextcloud, Owncloud, and many others) use PATCH with X-Update-Range header - see rust webdav_handler.

I believe that supporting these two approaches already covers many scenarios.