search

cern-fts / davix

High-performance file management over WebDAV / HTTP
GNU Lesser General Public License v2.1
84 stars 36 forks source link

Implement multipart streaming for DavPosix::write() #123

Closed abh3 closed 4 months ago

abh3 commented 4 months ago

This minimal pull request allows a client to specify that POSIX write requests should use multipart streaming via an envar DAVPOSIX_MPUPLOAD. If the envar is not set the original code path is used and uploads work just as before. This option is meantto be used by S3 proxy servers that cannot tolerate the creation of an intermediate file on POSIX write uploads.

abh3 commented 4 months ago

OK, you got the gist of why I was constrained. The architecture sort of sucks and short of duplicating all of the S3IO code, I had to add three more methods to the top of the http chain so that I could reuse the substantive code that was there. Had the original author of this code implemented multipart uploads correctly (which they did not) none of this would have been needed,

mpatrascoiu commented 4 months ago

You've also explained it very well that we don't have the effort to rewrite the codebase. We will leave the 3 new methods in the top HttpIOChain interface:

virtual std::string initiateMultipart();
virtual void writeFromBuffer(..);
virtual void commitChunks(..);

Please handle the indentation and work-around the _VOID chaining and it can go forward.

Cheers, Mihai

abh3 commented 4 months ago

All requests completed.

mpatrascoiu commented 4 months ago

Very well, thanks! Merging into devel.

Cheers, Mihai

abh3 commented 4 months ago

Hi Mihai,

Wei has been running stress tests and he found that a) no memory leaks, and b) no crashes. Good news indeed.

Andy

From: Mihai Patrascoiu @.> Sent: Friday, June 21, 2024 1:48 AM To: cern-fts/davix @.> Cc: Andrew Hanushevsky @.>; Author @.> Subject: Re: [cern-fts/davix] Implement multipart streaming for DavPosix::write() (PR #123)

Very well, thanks! Merging into devel.

Cheers, Mihai

— Reply to this email directly, view it on GitHubhttps://github.com/cern-fts/davix/pull/123#issuecomment-2182304523, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAUIW5ZTFH2QBNVV2D4RIBDZIPSGJAVCNFSM6AAAAABJPKKEMWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBSGMYDINJSGM. You are receiving this because you authored the thread.Message ID: @.***>