uqfoundation / pathos

parallel graph management and execution in heterogeneous computing
http://pathos.rtfd.io
Other
1.36k stars 91 forks source link

Shared Memory Objects #263

Closed AeroTH310 closed 1 year ago

AeroTH310 commented 1 year ago

It's not clear whether this library supports sharing memory between parallel processes. I've been able to find some threads here and there about it based on compatibility with the multiprocessing library, but I haven't gotten anything to work for my project yet. If there is any support, could we get an example in the documentation?

If there is no support, I would like to request it. It would seem like an extremely useful feature.

Of course, I will make a PR if I end up fixing this myself.

mmckerns commented 1 year ago

pathos itself doesn't have shared memory objects, however multiprocess does:

>>> import pathos
>>> import multiprocess as mp
>>> mp.Manager is pathos.helpers.mp.Manager
True
>>> mp.Array is pathos.helpers.mp.Array
True
>>> 

This gives you similar shared memory objects as in multiprocessing, where the documentation for how to use it should be virtually identical. If you are looking for a shared memory map, pathos doesn't have one. Please reopen if this doesn't answer your question.