configure complex_list for master and exec hosts

[delagoya]

Specifically I wanted to add the exclusive directive to the complex_list.

[delagoya]

best left for a plugin to do. Shall I close the issue?

[jtriley]

I've been meaning to pull the SGE code out into a separate plugin that's included with StarCluster.

This would allow users to configure it separately (via it's own [plugin] section) where could add an option for complex_list, for example, as well as queues to configure, etc. It would also allow users to run plugins before SGE is configured or just completely disable SGE altogether.

In this case I would likely want to add a setting like _enablequeue to the cluster config to remove the need to define a new plugin section just to get the default SGE install/config back. Does this sound reasonable?

[delagoya]

SGE plugin would make sense if you wanted to configure more than one scheduler (Condor, TORQUE, etc). In this case I would put in an setting in cluster config called "queue_engine" which defaults to SGE.

But this seems like a change better left for a major revision number. For this version I think the SGE support is the reason folks come to starcluster in the first place. I know it was for me.

I will see if I can get time to code a decent plugin to customize SGE installs with a few common options, starting with whether to remove master from exec host and add exclusive rights to the scheduler. Any other common options you would like to see?

[ank]

@delagoya, I just pushed a few plugins I find useful to save you little work. If you look at the 'extras' plugin it does some SGE config like setting the scheduler interval and adding the cluster user as the admin host. To remove the master as exec I think you could just add a 'qconf -de master' to my naive implementation.

https://github.com/bioteam/StarClusterPlugins

[delagoya]

Thanks Adam, will take a look at those.

[jtriley]

@ank excellent, thanks for forking StarClusterPlugins, this is precisely where I'm hoping to collect plugins from users. Eventually we should do a 0.1 release of StarClusterPlugins on pypi so that folks can install them easily. Feel free to submit a pull request whenever you're ready...

[jtriley]

@delagoya @ank the SGE setup code has now been moved to a new SGE plugin in the develop branch (see starcluster/plugins/sge.py) in case you're interested in adding new settings (via init kwargs) to customize the SGE install/configuration. Currently users will need to specify disable_queue=True in their config given that SGE is a default for now and then define the SGE plugin and specify it in your PLUGINS list in your config in order to customize the extra options. I'm happy to help if you have questions.