search

kothar / go-backblaze

A golang client for Backblaze's B2 storage
MIT License
95 stars 31 forks source link

Each upload is requesting a new b2_get_upload_url. #22

Closed antihax closed 6 years ago

antihax commented 6 years ago

It appears that each upload is requesting a new b2_get_upload_url and while I am attempting to concurrently upload a few million files, the progress is very slow.

Unless I am mistaken, the URL should be reused and they should be pooled. If an upload fails, the URL should be discarded from the pool. If they are all in use, a new one should be requested.

https://www.backblaze.com/b2/docs/b2_get_upload_url.html

kothar commented 6 years ago

I agree, they should be pooled. I think when the API was first released you only needed one upload URL, but this changed to require one per simultaneous upload, and the logic was not updated fully.

If you have time to attempt a patch I would gratefully accept it, otherwise I will look at this and the other pending issues when I get a chance to spend some time on maintenance.

antihax commented 6 years ago

I planned to take a look tonight. I may open a PR to track and keep it in WIP until it is finished.

On Wed, Aug 29, 2018, 11:22 Mike Houston notifications@github.com wrote:

I agree, they should be pooled. I think when the API was first released you only needed one upload URL, but this changed to require one per simultaneous upload, and the logic was not updated fully.

If you have time to attempt a patch I would gratefully accept it, otherwise I will look at this and the other pending issues when I get a chance to spend some time on maintenance.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/kothar/go-backblaze/issues/22#issuecomment-417013537, or mute the thread https://github.com/notifications/unsubscribe-auth/AGqJ23S_dxW69bSb-GNqwsNfWblr0mYlks5uVr_PgaJpZM4WR2Cg .

antihax commented 6 years ago

Had #23 in production overnight, significant drop in get upload URL requests.

kothar commented 6 years ago

Closing, patch very much appreciated 😄