search

roryk / ipython-cluster-helper

Tool to easily start up an IPython cluster on different schedulers.
148 stars 23 forks source link

Optional wait for all engines #28

Closed matthias-k closed 9 years ago

matthias-k commented 9 years ago

By default, ipython-cluster-helper waits only until the first engine is up. This is a useful defaul behaviour: If you are just using view.map, then you don't care if you cannot use all engines in the begin. However, in some cases you have to prepare your engines (for example, pushing some global data or compiling theano functions) before you can start processing data. In this cases one has to wait until all engines are available.

I implemened a new keyword argument wait_for_all_engines in ClusterView and cluster_view which controls whether you would like to wait for all engines. The default value is False, corresponding to the behaviour so far.

roryk commented 9 years ago

Thanks for all these nice contributions @matthias-k!