hep-gc / cloud-scheduler

Automatically boot VMs for your HTC jobs
http://cloudscheduler.org
Apache License 2.0
3 stars 0 forks source link

VM instances container could be optimized #186

Open AndreCharbonneau opened 13 years ago

AndreCharbonneau commented 13 years ago

The VM instances in the ICluster instance are kept in an array, and linear seach is used to access these VMs by id. Not a problem with a small number of VMs, but will put a considerable load on the CS with large number of VMs. A hash-based container would fix that issue. I propose we create a vm container class that will abstract out how the VM instances are stored and managed (similar to what was done for jobs). This container would use a hash table based approach; we can then further optimize if needed and the rest of the CS code will not be affected.

mhpx commented 13 years ago

Currently the ResourcePool class has its list of IClusters, each of which has its own list of VMs. Hashing both of these would be best as there are cases we have the VM but need to lookup which cluster it belongs to. This can easily be done with the clusteraddr(from VM)/network_address(cluster) property for the clusters. for VMs, if the hash is local to the cluster the VM.id is the easiest key. If we put in a single hash with all VMs it would need an aggregate key of sorts as IDs could overlap on nimbus clouds.

mhpx commented 12 years ago

We may be getting access to ~1000 boxes with 4-8 cores. Looks like it's time to revisit this issue.

igable commented 11 years ago

could we set a milestone for this?

igable commented 11 years ago

ping @AndreCharbonneau @mhpx