Boto2 unfortunately does not provide a bandwidth limiter for
S3 uploads. Instead, it will upload a completed backup as quickly
as possible, potentially consuming all available network bandwidth
and therefore impacting other applications.
This patch adds a very basic throttling mechanism for S3 uploads
by optionally hooking into the upload progress and determining
the current bandwidth. If it exceeds the designated maximum, the
upload thread will pause for a suitable amount of time (capped
at 3 seconds) before resuming.
While this is far from ideal, it is an easy to understand and
(from my experience) good enough method to protect other network
users from starvation.
Notice: The calculation happens per upload thread.
Boto2 unfortunately does not provide a bandwidth limiter for S3 uploads. Instead, it will upload a completed backup as quickly as possible, potentially consuming all available network bandwidth and therefore impacting other applications.
This patch adds a very basic throttling mechanism for S3 uploads by optionally hooking into the upload progress and determining the current bandwidth. If it exceeds the designated maximum, the upload thread will pause for a suitable amount of time (capped at 3 seconds) before resuming.
While this is far from ideal, it is an easy to understand and (from my experience) good enough method to protect other network users from starvation.
Notice: The calculation happens per upload thread.