search

uqfoundation / pathos

parallel graph management and execution in heterogeneous computing
http://pathos.rtfd.io
Other
1.36k stars 91 forks source link

Allow passing kwds to ProcessPool #252

Closed ddelange closed 1 year ago

ddelange commented 1 year ago

Hi 👋

I would like to propagate maxtasksperchild keyword, but for this I had to switch from pathos to multiprocess ref https://github.com/ddelange/mapply/pull/29.

This however decreases stability/cleanup of workers, so rather allow a pathos user to propagate it :)

ddelange commented 1 year ago

hi @mmckerns 👋 thanks for linking the open issues!

while we're at it, is there currently a way to propagate chunksize? since keyword arguments to imap are swallowed, I'm not sure wether passing it currently will be respected 🤔

mmckerns commented 1 year ago

I hate to admit it... but there's a PR (#198) that's been open for a while on chunksize and I haven't built a test set for it to see what will happen.

ddelange commented 1 year ago

it would be cool to merge both PRs, but agree that it would be cool to test (for both PRs) that behaviour changes when some kwarg is passed.

apart from tests, is there anything else to do before merging?

ddelange commented 1 year ago

added test

ddelange commented 1 year ago

@mmckerns anything from my side still?

mmckerns commented 1 year ago

This is good... it's on my shortlist to test and review.

ddelange commented 1 year ago

super! thanks for the review 👍

mmckerns commented 1 year ago

I also handled if someone passes processes in kwds. Seems to be done now. Thanks for the PR.