search

uqfoundation / pathos

parallel graph management and execution in heterogeneous computing
http://pathos.rtfd.io
Other
1.39k stars 89 forks source link

Timouts #123

Closed willgdjones closed 7 years ago

willgdjones commented 7 years ago

Hi there - is there some support for Pathos to timeout workers after e.g. 30 seconds?

Thanks!

mmckerns commented 7 years ago

Yes.

>>> import pathos
>>> p = pathos.pools.ProcessPool()
>>> f = lambda x:x*x
>>> res = p.amap(f, range(3))
>>> res.get(timeout=1)
[0, 1, 4]
>>> p.close()
>>> p.join()
willgdjones commented 7 years ago

Thanks for the quick reply @mmckerns ! One other thing, is it possible to specify a callback once a worker has completed a given job - e.g. to update a progress bar?

mmckerns commented 7 years ago

you should be able to make a progress bar by using res = Pool().imap(...) or uimap. They produce res as an iterator, and you can then iterate over res getting one result at a time as they come in. There is a good example of this I posted on SO, but I couldn't find the url for it quickly unfortunately.

willgdjones commented 7 years ago

Thanks @mmckerns! Pool or ProcessPool?

mmckerns commented 7 years ago

Oh, right... it should be ProcessPool above.

ortiz1093 commented 3 years ago

Hi @mmckerns, the timeout keyword in res.get() seems to timeout the entire pool. Is there a good way to timeout an individual worker and let the pool keep going?

mmckerns commented 3 years ago

@ortiz1093: Maybe this is not what you want to hear, but I think you'd want to use individual async pipes instead of a pool.

ortiz1093 commented 3 years ago

@mmckerns, no worries. I appreciate the response. I'm pretty new to multiprocessing, so I really didn't know what to use. I'll look into the async pipes. Thanks!