Distributed Calculations

[JamesPHoughton]

Optimization, Monte Carlo, etc. are all analyses that could benefit from running models in parallel.

We have a basic demonstration of parallel model fitting, where we fit the model independently to a lot of datasets: http://pysd-cookbook.readthedocs.org/en/latest/analyses/fitting/Massively_Parallel_Fitting.html

But we should show how these can be distributed across multiple computers. This is facilitated by the fact that pysd and its dependencies are pure-python, and can be pickled and distributed.

Here are a few resources for parallel computing in python:

• https://www.computer.org/csdl/proceedings/cluster/2013/9999/00/06702678.pdf
• https://sea.ucar.edu/sites/default/files/python_hpc.pdf

Distribution frameworks:

• multiprocessing
  • Part of standard python library
  • Runs easily in interactive mode
  • Basically limited to the cores on one host
• celery
• scoop
  • + Really clean API
  • - Doesn't play well with ipython/jupyter
• jug

[JamesPHoughton]

ipyparallel

Pros:

• Works well with ipython

Cons:

• Complicated setup

Resources:

• How to run a cluster on a single remote host: https://github.com/enthought/distarray/wiki/Running-a-simple-remote-cluster
• Running a cluster on multiple remote hosts: http://danielhnyk.cz/ssh-ipcluster-local-and-remote-engines/