search

uqfoundation / pathos

parallel graph management and execution in heterogeneous computing
http://pathos.rtfd.io
Other
1.38k stars 89 forks source link

Replacing multiprocessing.Process #178

Closed link-er closed 4 years ago

link-er commented 4 years ago

Hello. I am using multiprocessing to create separate Processes as forks from the main program and it works perfect in Ubuntu. Nevertheless, I cannot run it with Windows because of serialization problems. So I want to switch to pathos, but as I understand there is only an interface to work with ProcessPool. Can I directly create processes the way I was doing it already, without pool of processes? The import of Process class does not work smoothly. Thank you.

Coderx7 commented 4 years ago

I second this as well. this can come very handy!!

molinav commented 4 years ago

The custom multiprocessing with [dill] that pathos uses internally is also available as standalone package. It is called [multiprocess] and provides the same interface as the standard multiprocessing. If your use case does not require any additional features that are exclusive of pathos, using just [multiprocess] could be a solution.

In case you want to use it on Windows, be aware that [dill] has an issue between versions 0.2.9 and 0.3.1.1 that prevents module globals to be serialised on Windows and this issue is propagated to [multiprocess], so you will need to ensure that your installed [dill] is outside this range of versions (in the master branch of [dill] this issue is already patched).

[dill]: ../../dill [multiprocess]: ../../multiprocess

mmckerns commented 4 years ago

@molinav is correct. pathos has a thin wrapper on top of multiprocess, which already includes multiprocess.Process which serializes with dill. If you are on windows, you should make sure to use the latest dill from the repository (which includes this patch). A new release of dill is imminent.

mmckerns commented 4 years ago

Please reopen if this doesn't address your issue.