search

Azure / azure-storage-cpplite

Lite version of C++ Client Library for Microsoft Azure Storage
MIT License
25 stars 43 forks source link

Slow parallel upload using blob_client::upload_block_blob_from_buffer #96

Open BenjaminLanfer opened 3 years ago

BenjaminLanfer commented 3 years ago

When trying to upload especially smaller files using blob_client::upload_block_blob_from_buffer I find that upload speed is very slow (~500kbit/s). It appears that only a single parallel request is used although the maximum number of concurrent requests was set to 64.

From a cursory glance at the code, a rather large block size is chosen. That is, the minimum block size is set to azure::storage_lite::constants::default_block_size = 8 1024 1024, which is 8MB. Also, the grain size is set to around 4MB, where it is only 64kB in blob_client::download_blob_to_buffer.

Would it be feasible to set a smaller block size to allow for more blocks being uploaded in parallel even for smaller files?

Thanks for looking into this.

katmsft commented 3 years ago

The block size was originally chosen to enable LBB feature that can ultimately boost performance at the service side. This is considered a feature request to expose the option to specify the block size to a smaller value and is put to our backlog.

In the meantime, you can check the track 2 Storage SDK which has similar dependencies with CPPLite here and will be more actively maintained: https://github.com/Azure/azure-sdk-for-cpp/tree/master/sdk/storage

The block size can be customized in the mentioned track 2 SDK.