search

uqfoundation / pathos

parallel graph management and execution in heterogeneous computing
http://pathos.rtfd.io
Other
1.38k stars 89 forks source link

Is there any chunksize-like option from built-in processing module? #183

Closed hyungjunk closed 4 years ago

hyungjunk commented 4 years ago

Hi, I don't know if should be posted on issue.

I stumbled on pickling problem of built-in multiprocessing module, so I bypassed using pathos module which works fine with serialize/deserialize.

But I can't find option like chunksize in map method as it is in built-in multiprocessing module.

Is there any similar option like this?

mmckerns commented 4 years ago

Yes. If you use the pathos.pools that begin with an underscore (e.g. _ProcessPool, they have the traditional multiprocessing.Pool interface (i.e. have chunksize). You can also directly use the multiprocess package (which pathos is built on), and the Pool there has an interface identical to multiprocessing. All pools use dill serialization.

Reopen this issue if this doesn't answer your question.