agronholm / pythonfutures

Backport of the concurrent.futures package to Python 2.6 and 2.7
Other
232 stars 51 forks source link

ThreadPoolExecutor#map() doesn't start tasks until generator is used #28

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago
See this simple source: http://paste.pound-python.org/show/ZTWE9gxRpAX5qsgMGS2H/

On Python 3, the prints happen before the generator is used at all. On Python 
2, they don't happen and RuntimeError is raised when trying to iterate on the 
returned generator.

Python 2.7.8 on OS X 10.9 with futures 2.1.6.

Original issue reported on code.google.com by remirampin@gmail.com on 30 Aug 2014 at 9:23

GoogleCodeExporter commented 9 years ago
Interestingly enough, if I use this project instead of the stdlib on Python 3, 
I get the same RuntimeError. Is this project still maintained at all?

Original comment by remirampin@gmail.com on 30 Aug 2014 at 11:17

GoogleCodeExporter commented 9 years ago
In Python 3's version, the map function is not an iterator, it returns an 
iterator. This way processing all the futures are already submitted by the time 
the with statement exits.

https://hg.python.org/cpython/file/334c01aa7f93/Lib/concurrent/futures/_base.py#
l552

However, I'm not sure your example falls into proper use. It's kind of like 
opening a database connection, sending a query, closing the connection, and 
then trying to read the results. You should grab the results why the executor 
is still running.

Original comment by math...@closetwork.org on 3 Dec 2014 at 5:34

GoogleCodeExporter commented 9 years ago
The problem is not that an iterator is returned, it's that the futures don't 
get submitted at all until I do list(result).

Original comment by remirampin@gmail.com on 3 Dec 2014 at 7:25

GoogleCodeExporter commented 9 years ago
Here's another example, since you seem confused: 
http://paste.pound-python.org/show/w5g7O1rUASuxkB17OKCs/

I'd very much like this library to work again, some of my stuff depends on it.

Original comment by remirampin@gmail.com on 19 Dec 2014 at 11:07

agronholm commented 9 years ago

Fixed in bf9f88c.