uqfoundation / pathos

parallel graph management and execution in heterogeneous computing
http://pathos.rtfd.io
Other
1.39k stars 89 forks source link

No timeout arg in the next() method of the iterator returned by imap() #106

Closed mxndrwgrdnr closed 7 years ago

mxndrwgrdnr commented 7 years ago

The built-in multiprocessing library provides a timeout argument for next() method of the iterator object returned by Pool.imap(). This is really handy for messy subprocesses. For example, I am using imap to spawn Chrome instances using Selenium, which when working with a large number of iterations are prone to freezing up. I'm using Pathos b/c my parallelized function is a class method, which the built-in library cannot handle. But unless Pathos supports a similar timeout function, I'll probably have to refactor my code to use the built-in library. Any ideas?

mmckerns commented 7 years ago

It is supported... however, it's just not yet supported in the "unified" interface.

>>> import pathos
>>> p = pathos.helpers.mp.Pool()
>>> res = p.imap(lambda x:x**2, range(4))
>>> res.next(timeout=1)
0
>>> res.next(timeout=1)
1
>>> res.next(timeout=1)
4
>>> res.next(timeout=1)
9
>>> p.close()
>>> p.join()

The above library (pathos.helpers.mp), is a direct handle to the multiprocess library, which was split off from pathos into it's own package.

If this isn't sufficient of an answer, please reopen the ticket.