Make an autoscaling plugin

I plan to write an autoscaling plugin system.

It will be invoked like so:

python manage.py autoscaler

And it will then look at the plugin configuration and work somewhat like my comments mentioned in this thread:

https://talk.vespene.io/t/slurm-isolation/

It should be usable to execute anything, but a trivial shell executor that can easily pass arguments to something like terraform or a custom shell script would be the easiest thing to write.

I'm not positive exactly what configuration will look like, but building this will also involve a new docs chapter.

The common use case this is intended to cover is the classic "AWS autoscaling" worker case.

To do this, we can also add a new flag to the worker, that allows it to run one and only one job and then exit, which could make some of this easier.

(We could also add a flag to the worker so it doesn't try to process the organization).

Configuration of autoscaling workers would be somewhat of an advanced topic, requiring construction of a good worker image preconfigured to speak to the DB, but in all this seems like it would be a very easy system to maintain. By suggesting tools like terraform and making it pluggable, we can avoid maintaining any cloud specific plugins.

vespene-io / _old_vespene

Make an autoscaling plugin #80