search

google / skicka

Command-line utility for working with Google Drive. Join the mailing list at https://groups.google.com/forum/#!forum/skicka-users.
Apache License 2.0
1.3k stars 119 forks source link

Files are completely buffered in memory #13

Closed agoode closed 9 years ago

agoode commented 10 years ago

If I am uploading large files, the files are buffered in memory completely. There should be a way to stream to the server. This might need to be fixed in the google API library itself.

mmp commented 10 years ago

Unfortunately I'm pretty sure this will require fixing the Google Drive APIs (or bypassing them and doing the POST directly through Go's http library. (Assuming that doesn't end up buffering things...))--a while ago I did some experiments trying to rate-limit uploads by implementing an io.Reader that did rate-limiting and then using that for the Files.Update() call; unfortunately, the Drive API implementation ended up draining that locally into a copy before starting the upload.

But this is absolutely worth fixing!

mmp commented 9 years ago

The rate-limited upload commit reduces the number of full copies of the file in memory by 1 at least. This also lays framework for completely avoiding buffering in memory, though I'm going to first try to get chunked uploads working, which should be more robust to auth tokens expiring in the middle of large uploads, etc.

mmp commented 9 years ago

This is fixed with 8a54979477 and 032f6b4. (Woo woo).