search

pplu / aws-sdk-perl

A community AWS SDK for Perl Programmers
Other
171 stars 94 forks source link

S3 Streaming Upload? #91

Open frioux opened 8 years ago

frioux commented 8 years ago

Ok so I'm nearing the end of my "automultipart" thing and wanted to ping you to see if you'd also be interested in an additional patch to allow passing a filehandle as the arg to Body, so that the full body never needs to reside in memory during a possibly large upload. Seems like a straight win to me, but maybe you know something about S3 that I don't that would make this impossible?

pplu commented 8 years ago

Hi,

I'm interested in any type of patch that adds value to Paws, so if you feel like the FH stuff doesn't add the sufficient value for you: don't bother (it's not a prerequisite to accept your automulipart stuff). If you think about it, the chunks will have to reside in memory one time or the other, so even if we're still handling the chunks in strings, we should be handling a manageable size and number of chunks at any time.

I don't know of anything in S3 that would make it impossible. There are a couple of points that I haven't stiched together yet (as the FH stuff has just been a thought in the back of my mind for some time): Can the body of a request be a filehandle in HTTP::Tiny, LWP, Furl, Mojo::UserAgent? If so, how can you tell the HTTP client(s) to just send X bytes of the fh (for multipart uploading)? Also you have to calculate a hash of the content in the signing phase (https://github.com/pplu/aws-sdk-perl/blob/c3decfecb6d9ce374ba73f0126ac9ffe6152eda4/lib/Paws/Net/S3Signature.pm). I think the hasher supports filehandle interface, but that means you have to rewind the descriptor to the original position before handing off to the http client to send a chunk.

Happy coding :smile:

frioux commented 8 years ago

Yeah, the signing is exactly what I figured would be problematic. It's annoying, but possibly still worth it to avoid loading large (think 500 meg) chunks into memory, since perl will never release that back to the OS. I still would like to do this at some point, but the automultipart is more important right now.