search

skshetry / webdav4

WebDAV client library with a fsspec based filesystem and a CLI.
https://skshetry.github.io/webdav4
MIT License
61 stars 17 forks source link

Upload of large files is never chunked #168

Closed Ph0tonic closed 2 months ago

Ph0tonic commented 1 year ago

Hi,

When uploading large files like 100 Mo or bigger ones (1 To), these are never chunked. I would expect the client to offer a way to enforce chunk encoding or automatically chunks large files. An alternative to the current method which tries to always avoid chunk encoding would be to use chunk encoding when a chunk_size is provided (see #167). What we could also do is add an extra flag to enforce chunks. Thanks,

skshetry commented 1 year ago

Hi @Ph0tonic. Could you please elaborate why you need to chunk? What issues do you run into?

The reason why we don't prefer chunked-encoding is that not a lot of servers support chunked encoding in PUT. This is mostly older servers, but we do bump into them time to time. See https://github.com/iterative/dvc/issues/4796.

Ph0tonic commented 1 year ago

Hi, So I will have to test it again, but we had some trouble going through a proxy with non-chunked encoding. I understand your point of view and I will check again with our new set-up to see if we still have this issue. Thanks for your answer

skshetry commented 1 year ago

Any updates?