search

uqfoundation / pathos

parallel graph management and execution in heterogeneous computing
http://pathos.rtfd.io
Other
1.38k stars 89 forks source link

ProcessPool vs pathos.helpers.mp.Pool #204

Closed visrra closed 3 years ago

visrra commented 3 years ago

Are there any advantages of ProcessPool compared to pathos.helpers.mp.Pool? The latter seems to be able to handle lambda functions, and it can also display logging outputs of tasks.

mmckerns commented 3 years ago

There are several differences, but underneath they are the same. pathos.pools.ProcessPool is built on top of pathos.pools._ProcessPool (same object as in pathos.helpers). The latter exactly replicates the multiprocessing interface, while the former conforms to an interface that is uniform across several backends. The ProcessPool can take several *args, is globally cached, and can be reopened once closed. The_ProcessPool gives you whatever multiprocess.Pool provides. Both cases can handle lambda functions, and should provide exactly the same level of serialization.

visrra commented 3 years ago

Thanks @mmckerns