search

uqfoundation / pathos

parallel graph management and execution in heterogeneous computing
http://pathos.rtfd.io
Other
1.38k stars 89 forks source link

pass the chunksize parameter to multiprocessing.Pool().map() #198

Closed x4base closed 1 year ago

x4base commented 3 years ago

Hi,

I saw your comment here: https://stackoverflow.com/a/55615340/6804110, and I see you didn't put in the chunksize parameter by design. But I think passing the kwds directly to multiprocessing.Pool().map() might solve the problem, because the only keyword argument left is chunksize. And by using kwds, the users can choose whether they want to pass the chunksize parameter. I can also change it to map(self, f, *args, chunksize=None, **kwds) if you prefer this way.

Does it make sense to you? Will it cause any side effect?

mmckerns commented 3 years ago

Sorry for the slow reply. I think this is reasonable. However, your PR is incomplete -- chunksize is used in other maps as well. I'd also pop chunksize out of kwds, to minimize potentially unwanted effects. I may just implement it and void this PR, depending on availability.

ddelange commented 1 year ago

@mmckerns opened #253 including tests that would fail if chunksize were not propagated

mmckerns commented 1 year ago

closed by https://github.com/uqfoundation/pathos/commit/d7e9e0b17bb00bc574faccf014ad6f0a476fc95b