Should we switch from Scoop to IPyParallel?

[rgerkin]

@russelljjarvis

[russelljjarvis]

SCOOP vs Ipyparallel.

Strengths Weaknesses Opportunities Threats:

Arguments in favor of scoop:

• Already in use

• was designed for use with DEAP, and has tight DEAP integration in code conventions.

• Uses zeromq (Z0Q) natively which is more communication speed efficient than MPI

• Allows recursive calling of futures.map facilitating a tree shaped cluster topology that should result in a bigger spreading of code onto CPUs (greater utilization of parallel resources).

• Note I don't necessarily know that ipython lacks this feature, and if it does, its possible that it won't always.

Scoop draw backs:

• ipython unfriendly invocation patterns, limited documentation.

• verbose and unsugared syntax, causes obfuscation to casual code readers.

ipython-parallel advantages.

• great ongoing documentation and development.

• full of syntactic sugar.

• should speed up development, and debugging.

ipython-parallel dis-advantages.

• I don't yet understand its ipypthon parallels error messages.

• Perhaps there might be ways of speeding up the end product, such as converting notebooks back to regular ipython code, and then interpretating ipython to cython as much as possible.

• Seems to prefer that the ipcontoller, and the ip engines (exist in two seperate docker containers that are launched in parallel and communicate to each other via ssh tunneling, as several authors have noted .. Although I don't know the reasons for this preference, and if it really has to be so.

https://github.com/0x0L/jupyter-cluster https://github.com/ActivisionGameScience/ipyparallel-mesos/ https://github.com/ipython/ipyparallel/issues/51

• Given the above point, that it prefers two separate dedicated docker containers that are run simultansuosly and communicate with each other:

** It seems like the resulting nb, should not be user friendly to run, and would therefore be in violation of notebook philosophy.

** Mapping docker container to singularity image would be clumsy, since the singularity image would need a docker daemon, and two containers to execute. From experience, I know that docker in singularity is highly discorouged and not very good ... although this could change quickly?

• If ipythonparallel does lack a recursive calling of view.map_sync, then I may have to do a medium effort algorithm re-write.

• 0MQ as a communicator is not default, but can be enabled with a flag

Apache-Spark,

another parallel framework for python, I don't really understand but there seems to be a lot of community and support for it.

Future directions.

It could make sense to do try to do all future analysis, and plotting development work of the optimization outputs in parallel ipython notebooks.

But to also make an initial notebook that makes calls to the os.system('python -m scoop filename.py') via ! ipy magic, do checkingpointing of the namespace variables created in these files and drag the checkpointed names into ipython notebook, maybe using the module query

I can also notebookerise this code by placing it in markdown cells

Needed to make better judgements:

• A single container implementation of ipyparallels and ipcluster/ipcontroller that works properly and can be elaborated.

This might be such a thing: https://github.com/vishnu2kmohan/ipyparallel-docker

• A jupyter notebook that can run scoop by feigning command line arguments normally fed in via argparse, achieved alternatively with a pickle load of conventional scoop parameters that take by default launch scoop using all available local resources.

** A methods for A translating ipython notebook code blocks, into not just pure python code, but actually pure python instructions.

** A modification of scoop.futures.launcher init constructor, such that it expects not the file of python code executable.py, but instead can except a long string or list of strings, that are executable python instructions ie a rewrite of launcher, such that it can act on and satisfy exec('instructions') as opposed to execfile('file.py').

This might sound simple but its probably deceptively a lot of work.

ipython parallel can also work with numba and Just In time compiled numba code.

http://numba.pydata.org/numba-doc/0.11/prange.html

[russelljjarvis]

After doing a lot of the work associated with switching, I now believe that the answer is yes, we should switch to ipyparallel.

The two main reasons are: Better integration with documentation via, notebooking, and also because notebooks have checkpointing, which can save time on big runs.

I am getting the sense that ipyparallel might be slower than scoop, however I am unsure if things have to be this way, and if its possible to set the backend as ZeroMQ (0MQ), which might even be the default in future releases.

I also do not know exactly how to get read the stack traces off workers that fail. I am guessing you can set a flag that creates a verbose log file with each run but I am unsure how yet.

If stderr suddenly becomes accessible, then at least development (including documentation development time) might become faster via ipyparallel, at least because of checkpointing.

[rgerkin]

Sounds good.

[russelljjarvis]

Dask like ipyparallel supports a simple invocation of ipython, and it supports nested calls to map_reduce in a way that truly flattens out an otherwise sequential method call flow.

ipyparallel supports integration with dask, and dask as a backend. Dask has more mature documentation, in the sense that it is very realistic about performance improvements, and concerned that the code will not result in good enough utilization and efficiency of CPUs, as compared to serial execution that leverages numba and jit compilation of iteration routines and or use of cython for optimization.

https://gist.github.com/mrocklin/dc7fd426a6e405fef275

dask also better support for hdf5 and pandas, but possibly less syntactic sugar.

from ipyparallel import Client
c = Client()  # connect to IPyParallel cluster
e = c.become_dask()  # start dask on top of IPyParallel

Since ipythonparallel seems to be dask aware, hopefully, some of dasks features will converge into ipyparallel.

[russelljjarvis]

A pure dask implementation might allow for nested calls to map_reduce.

Also python-intel distributed has since been released, but it comes in binaries only, so it is not really open source I guess. https://software.intel.com/en-us/distribution-for-python/features

[russelljjarvis]

I actually think we should switch to dask sometime in the future. This is based on what the primary author of ipyparallel recommends on stackexchange.

Dask is just as ipython friendly as ipyparallel, however I think it's also a more versatile tool, with more extensive documentation.

https://stackoverflow.com/questions/37998484/how-to-efficiently-chain-ipyparallel-tasks-and-pass-intermediate-results-to-engi Qoute from that stack exchange thread: " Option 2: dask.distributed

Full disclosure: I'm the primary author of IPython Parallel, about to suggest that you use a different tool.

It is possible to pass results from one task to another via engine namespaces and DAG dependencies in IPython parallel, but honestly, if your workflow looks like this, you should consider using dask distributed, which is designed specifically for this kind of computation graph. If you are already comfortable and familiar with IPython parallel, getting started with dask should not be too much of a burden. "

[rgerkin]

I'm fine with it, but let's delay it until after the next release of NeuronUnit. We really need to focus on code coverage, passing tests, and readable, documented code. If IPyParallel passes simple tests but fails with very large optimizations, that is OK for the next release.

[russelljjarvis]

I accidentally switched first.

Reasons:

• Dask has great support for pandas/numpy. See here:

http://dask.pydata.org/en/latest/dataframe.html

• No clunky invocation, with calls to external shell and wait, with hit and miss cluster building.

import statements such as import dask etc are sufficient for creating a local cluster like environment.

• noticeably better performance detectable on a human level.

• A simple parallel function mapping API via dask bags: http://dask.pydata.org/en/doc-test-build/bag.html

• debugging of hosts on local stdout achievable with:

With the single machine schedulers, use the rerun_exceptions_locally=True keyword.

x.compute(rerun_exceptions_locally=True)

On the distributed scheduler use the recreate_error_locally method on anything that contains Futures :

From: https://dask.pydata.org/en/latest/debugging.html

[russelljjarvis]

@HamiltonWang.

If you cast a numpy array to type

pop_bag = dask.db.from_sequence(pop)

you can use:

pop=list(pop_bag.map(method_type).compute())