search

uqfoundation / pathos

parallel graph management and execution in heterogeneous computing
http://pathos.rtfd.io
Other
1.39k stars 89 forks source link

add support for concurrent.futures #90

Open mmckerns opened 8 years ago

mmckerns commented 8 years ago

the pathos interface is nearly a match to concurrent.futures, and it seems to be preferred over multiprocessing as the standard.

PawelTroka commented 6 years ago

@mmckerns Any updates on this?

mmckerns commented 6 years ago

No, I haven't had time for it. In theory, it should be an easy PR -- however, I'm short on time at the moment.

evanbiederstedt commented 4 years ago

+1

This would be very very useful

colibri17 commented 4 years ago

+1 Agree

h3jia commented 4 years ago

+1

matthewyangcs commented 1 year ago

+1 please!

ddelange commented 1 year ago

fwiw, in case you're in the asyncio world and you're in need of dill, this gist might be useful to you (particularly the AioPool part): https://gist.github.com/ddelange/643fbb791b398783c04d1ceb90102163

See also the PR I liked above in 2021. It's not a drop in replacement of ProcessPoolExecutor, but the gist mostly gets the job done for me.

mmckerns commented 1 year ago

I just had this come up in a funded project yesterday, so this may get prioritized...

ddelange commented 1 year ago

@mmckerns that's cool to hear! the unresolved comment might be relevant for the implementation

rachtsingh commented 2 months ago

One important difference between multiprocessing.Pool and concurrent.futures.ProcessPoolExecutor is that the former hangs if a worker process is killed. Does the pathos/multiprocess project have a plan to handle that behavior (or is it already handled in multiprocess.Pool)?

As an alternative for those looking, it looks like the joblib/loky project does try to handle this? I'll give it a shot.

ddelange commented 2 months ago

I think that's https://github.com/uqfoundation/pathos/issues/265