Improve performance for upload/download of small files

[paya-cz]

I have benchmarked the SDK by uploading/downloading a total of 64 MiB in various-sized files and these are the results:

4 KiB chunks, total of 16 384 files, max concurrency 50
Upload: 1 hour 5 min 49 sec (16.6 KiB/s)
Download: 41 min 56 sec (26 KiB/s)

16 KiB chunks, total of 4 096 files, max concurrency 50
Upload: 15 min 50 sec (69 KiB/s)
Download: 9 min 37 sec (114 KiB/s)

64 KiB chunks, total of 1 024 files, max concurrency 50
Upload: 5 min 5 sec (215 KiB/s)
Download: 2 min 25 sec (452 KiB/s)

128 KiB chunks, total of 512 files, max concurrency 50
Upload: 2 min 26 sec (449 KiB/s)
Download: 1 min 13 sec (895 KiB/s)

256 KiB chunks, total of 256 files, max concurrency 50
Upload: 1 min 31 sec (718 KiB/s)
Download: 36 sec (1.8 MiB/s)

512 KiB chunks, total of 128 files, max concurrency 50
Upload: 58 sec (1.1 MiB/s)
Download: 19 sec (3.4 MiB/s)

1 MiB chunks, total of 64 files, max concurrency 50
Upload: 39 sec (1.6 MiB/s)
Download: 12 sec (5.4 MiB/s)

64 MiB chunks, total of 1 file, max concurrency 1
Upload: 36 sec (1.8 MiB/s)
Download: 9 sec (7.0 MiB/s)

Does the SDK make proper use of HTTP pipelining? The upload of 4 KiB chunks is really slow. Is there anything there can be done about this client-side? Maybe this could be solved by adding an API which allows for uploading a batch of files at once? Something like uploading a big .zip file and telling the OneDrive service to extract it in the destination upon transfer completion.

[daboxu]

@paya-cz thanks for the benchmark info, it is very useful. Unfortunately we don't have any API support uploading batch of files at once. I agree with you the uploading right now is not efficient and it is due to the bottleneck at the API side so there is no improvement I can imagine on the client side. We are working on improving the API uploading performance and I will upgrade to it once the API is updated.