ipcluster can remain open in some circumstances

[gregcaporaso]

From @wdwvt1 on February 27, 2016 7:40

If a user deletes the output folder that has been created before all the jobs launched by IPcluster have finished, the cluster will fail to shut itself down. This could cause serious resource leaks.

_Copied from original issue: biota/sourcetracker2internal#23

[wdwvt1]

@gregcaporaso noted that we need to close the cluster if gibbs or gibbs_loo fail.

[wdwvt1]

@wasade mentioned using the atexit functionality here

[wasade]

It might be worth replacing ipcluster with joblib, although I don't know how coupled the codebase is at this time. Joblib is super simple, performant, what scikit-learn uses, and can use a ipython cluster backend anyway if it is necessary to span multiple nodes

[johnchase]

@wdwvt1 this should be fixed by #117