search

datacratic / StarCluster

StarCluster is a utility for creating and managing computing clusters hosted on Amazon's Elastic Compute Cloud (EC2).
http://star.mit.edu/cluster
GNU Lesser General Public License v3.0
37 stars 13 forks source link

reduce add_node aggressiveness by accounting for avg job duration #46

Closed FinchPowers closed 8 years ago

FinchPowers commented 8 years ago

If the cluster has a running average of 15 minutes and sees 10 jobs waiting, will no longer add 10 nodes but 3 instead.

FinchPowers commented 8 years ago

Not too good after all. I looked at our production cluster and the job duration average is currently 8 seconds. It means that to add 2 nodes we would need to have at least 451 jobs in the queue and that our current slot usage would be maxed.

As this is low value, I will drop it entirely for now.