search

cern-fts / davix

High-performance file management over WebDAV / HTTP
GNU Lesser General Public License v2.1
84 stars 36 forks source link

Swift write from provider #62

Closed ashleylst closed 3 years ago

ashleylst commented 3 years ago

Add uploading files function for Swift, with multipart upload considered.

One can use 'davix-put local/file/path swifts://.. --ostoken $token --osprojectid $id' to upload local file to Swift, and use '-r' option to upload a directory of files to Swift. One can use fragment '#forceMultipart' to make use of multipart upload supported by Swift.

Multipart upload is used for files > 512MB. In this case, a large file is uploaded in a set of segments of maximum size 256MB. The default maximum segment number in one multipart manifest in Swift is 1000, hence, large files that exceed the 1000 limit in segments will be uploaded in an inline manner (where the file is represented by manifests within one manifest).

ashleylst commented 3 years ago

Thanks for the comment, I did not realize I was using an additional variable, as I focused on seeing if it is possible to commit chunks in an inline manner. Now it should be fine.