search

startover / pythonfutures

Automatically exported from code.google.com/p/pythonfutures
Other
0 stars 0 forks source link

map is greedy #20

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
map includes this line:

  fs = [self.submit(fn, *args) for args in zip(*iterables)]

That means no futures code gets executed until all the iterables have been 
completely consumed.  If each iteration of an iterable can take quite some 
time, that means a long wait before any futures even starts.

It also means a humungous list is built up if the iterables returns many items. 
 For example if iterable was xrange(2000000) then that fs list is going to be 2 
million items long even if num_workers was something like 8.

A generator expression should be used instead.

Original issue reported on code.google.com by rogerbinns on 7 Jun 2013 at 2:01

GoogleCodeExporter commented 9 years ago 
I ran in the same problem, I kept running out of memory.

Including a patch to make map non-greedy. The number of outstanding futures is 
configurable.

Original comment by math...@closetwork.org on 3 Dec 2014 at 5:11

Attachments: