Add Multithreaded Large File Upload

[mitchellhuang]

Changes

• Add multithreaded large file upload via B2FileList.upload_large_file
• Add B2Connector.upload_part for uploading parts of a large file
• Add recommended_part_size to B2Connector._authorize
• Update README

For multithreading, we use multiprocessing.dummy to import a ThreadPool and perform a pool.map of b2_get_upload_part_url and b2_upload_part across get_part_ranges(content_length, part_size). part_size defaults to recommended_part_size and num_threads defaults to 4.

Fixes https://github.com/sibblegp/b2blaze/issues/7.

[sibblegp]

Looking good. I'll merge and push a new version when I'm at the right computer! Thank you!

[mitchellhuang]

Cool, just squashed some miscellaneous commits so master will look cleaner after the merge. Thanks for the great library - code was easy to read and modify!

[sibblegp]

Related to issue #7

[sibblegp]

It's standard in library examples not to mention all options. For example, look at the examples for "requests".

Sincerely, George P. Sibble

On Sat, Jul 21, 2018 at 3:50 PM, Mitchell Huang < notifications@github.com > wrote:

@mitchellhuang commented on this pull request.

In README. md ( https://github.com/sibblegp/b2blaze/pull/14#discussion_r204223556 ) :

bucket.files.upload(contents=text_file, file_name='folder/hello.txt') ```
-NOTE: Reading the file is optional. You don't have to call .read() and
instead can send the file directly to contents. This should save some
memory. +NOTE: Reading the file is optional. You don't have to call
`.read()` and instead can send the file directly to contents. This should
save some memory. + +#### Upload a Large File + +```python +large_file =
open('large_file.bin', 'rb') +new_file =
bucket.files.upload_large_file(contents=large_file,
file_name='folder/large_file.bin', + part_size=100*1000*1000,
num_threads=4)

Yeah but then people won’t know you can specify a part size right?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub ( https://github.com/sibblegp/b2blaze/pull/14#discussion_r204223556 ) , or mute the thread ( https://github.com/notifications/unsubscribe-auth/AA-pqtJuvmnB_al0CRmTZ10uz5_QaSI-ks5uI7A9gaJpZM4VZRsh ).

[mitchellhuang]

I made some significant enhancements in the latest commit:

• The sha1 hash is now calculated on the fly during .read() and appended at the end of the request body to B2 when possible.
• Both upload and upload_part now buffer over .read() directly to B2.
• Added optional progress_listener argument to upload and upload_large_file to check the progress of an upload.
• Removed part_size argument to upload_large_file from README.md

[sibblegp]

Very cool. I’ll take a look asap.

Sincerely, George P. Sibble

---

From: Mitchell Huang notifications@github.com Sent: Monday, July 23, 2018 12:06 AM To: sibblegp/b2blaze Cc: George Sibble; Comment Subject: Re: [sibblegp/b2blaze] Add Multithreaded Large File Upload (#14)

I made some significant enhancements in the latest commit:

• The sha1 hash is now calculated on the fly during .read() and appended at the end of the request body to B2 when possible.
• Both upload and upload_part now buffer over .read() directly to B2. This makes it so you no longer need part_size * num_threads amount of memory.
• Added optional progress_listener argument to upload and upload_large_file to check the progress of an upload.
• Removed part_size argument to upload_large_file from README.md

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/sibblegp/b2blaze/pull/14#issuecomment-406960739, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AA-pqtLsnOJi0xrOxvpLAwse-3YdKhcWks5uJXXygaJpZM4VZRsh.

[sibblegp]

Could you please add unit tests for this?

[mitchellhuang]

Yeah sure i'll try to later this week, busy with some other stuff rn