Call SCOOP from within Python

[inJeans]

Is there a way to distribute a Python function using SCOOP from within Python?

The current workflow is to call your python function using python -m scoop ... from the command line.

Can I distribute a particular function in my python script without modifying the command line interface?

I am trying to modify an existing script and I do not want to change the command line interface.

Thanks

[anandtrex]

+1. This would also help to control when the scoop worker processes are spun up and spun down programmatically without them having to run throughout the entire duration of the main program. (For example if only one part of the code uses scoop for remote execution)

[russelljjarvis]

+1 This would also help for making ipython notebooks of DEAP optimizations.

[Dadle88]

Would love to see movement on this issue to run this in Jupyter

[russelljjarvis]

Yes me too. I love scoops capacity to do futures.map inside futures.map, but I have largely moved onto dask, mainly due to the issue of invoking dask simply by a module load, as opposed to having to launch with a special command line argument.

[jiangchao123]

@russelljjarvis excuse, can your code solve the problem?

[russelljjarvis]

No, I learned to use dask.bag map instead. Dask bag map works really well. https://dask.pydata.org/en/latest/

[jiangchao123]

thanks~