Feature request: dynamic threads

It would be really neat if s3backer were able to adjust the number of active 
threads based on performance.

Here's my thinking...

When too many threads have been configured for the available upload bandwidth, 
packets start to get lost and s3backer starts to see operation timeouts. 
Suppose s3backer took advantage of this as follows:

* Allow the user to configure a minimum and maximum number of active threads.
* Start out using the maximum configured number of threads.
* Each time an operation timeout happens, decrease the active thread count by 
1, unless it's already at the minimum.
* Whenever a certain number of writes to S3 occurs without any timeouts, 
increase the active thread count by 1 unless it's already at the maximum.
* Log the active thread count each time it is decreased or increased, so that 
the user can determine from his logs the optimal number of threads to use.

With this approach, I believe that s3backer will hover most of the time around 
the optimal active thread count, with occasional short-lived detours lower or 
higher.

I took a stab at implementing this but the code is sufficiently complex that I 
didn't feel like I could do it justice in the time I have available. It would 
probably be easier for the guy who wrote the code. ;-)

Original issue reported on code.google.com by jikam...@gmail.com on 19 Oct 2010 at 4:25

andriybobyr / s3backer

Feature request: dynamic threads #22