Alas, with pure python our options for parallelism are limited.
I have implemented 4 different ways of performing this parallelism and compared them on performance, complexity, and flexibility. What I have settled on is a particular way of using the python multiprocessing library.
The biggest downside to this method is that it requires anything that appears in Push programs and evaluators to be serializable with the built-in pickle serialization. This means no lambda functions allowed.
Alas, with pure python our options for parallelism are limited.
I have implemented 4 different ways of performing this parallelism and compared them on performance, complexity, and flexibility. What I have settled on is a particular way of using the python
multiprocessinglibrary.The biggest downside to this method is that it requires anything that appears in Push programs and evaluators to be serializable with the built-in
pickleserialization. This means no lambda functions allowed.