map is greedy

[GoogleCodeExporter]

map includes this line:

  fs = [self.submit(fn, *args) for args in zip(*iterables)]

That means no futures code gets executed until all the iterables have been 
completely consumed.  If each iteration of an iterable can take quite some 
time, that means a long wait before any futures even starts.

It also means a humungous list is built up if the iterables returns many items. 
 For example if iterable was xrange(2000000) then that fs list is going to be 2 
million items long even if num_workers was something like 8.

A generator expression should be used instead.

Original issue reported on code.google.com by rogerbinns on 7 Jun 2013 at 2:01

[GoogleCodeExporter]

I ran in the same problem, I kept running out of memory.

Including a patch to make map non-greedy. The number of outstanding futures is 
configurable.

Original comment by math...@closetwork.org on 3 Dec 2014 at 5:11

Attachments:

• patch.map.nongreedy.txt

[agronholm]

Fixed in fdbc9c3.

[mathieulongtin]

Humbled by your much simpler fix...