search

bernd-wechner / Degoo

CLI tool(s) for working with Degoo cloud storage
Other
143 stars 41 forks source link

Issue uploading large files #9

Closed MDKPredator closed 3 years ago

MDKPredator commented 3 years ago

When I try to upload large files, a MemoryError exception occurs when executing request.post. I have tried with a 1Gb file. I managed to fix it by changing this:

# Perform the upload
# TODO: Can we get a progress bar on the this? Web app has one.
response = requests.post(BaseURL, files=parts, headers=heads)

By this:

from requests_toolbelt import MultipartEncoder

multipart = MultipartEncoder(fields=dict(parts))
heads['content-type'] = multipart.content_type
response = requests.post(BaseURL, data=multipart, headers=heads)

Greetings.

bernd-wechner commented 3 years ago

Thanks heaps. Care to PR it? I wonder if we can get an upload progress bar happening this way?

Guess I never ran into memory issues as I have a full 32GB memory outfit ;-). I've certainly uploaded 1GB file as a test. But thanks for this!

MDKPredator commented 3 years ago

It may be possible to use the progress bar. I will check it before launching the PR

bernd-wechner commented 3 years ago

Thanks again. There's an existing model for a nice terminal base progressbar of course in the wget implementation already:

https://github.com/bernd-wechner/python3-wget

And I imagined implementing a wput package with a similar signature for consistency, whic once I'd worked out how to do the upload with a progress feedback I'd probably do by forking the wget and simply changing the internals to upload rather than download and renaming it and adjusting all the package data.

But that's just how I imagined implementing it.

MDKPredator commented 3 years ago

Thanks to you for your work.

I have created the PR, I forgot to add that I have reorganized the imports with IntelliJ

bernd-wechner commented 3 years ago

Re: imports, no worries, am using Codacy on another repo of mine may add it here. Offers nice automatic feedback.

Merged the change and will get around some to playing a round with it too ;-). For now squeezing comments like this into spare moments between other commitments.