[Errno 32] Broken pipe When Mapping Too Many Values

[srkunze]

First of all, thanks for that great backport. I am using it for the xfork package to support the 2.7 branch.

Unfortunately, there is something strange happening with this futures distribution which works perfectly fine with python3.4

from concurrent.futures import ProcessPoolExecutor

def calc(n):
    with ProcessPoolExecutor() as pool:
        results = pool.map(term, range(n))
        return sum(results)

def term(x):
    return x

print(calc(5000))

/usr/bin/python2.7 calc.py
12497500
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 266, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe

[kadnan]

Did you find solution?

[srkunze]

@kadnan Not yet. Run short of time for now. Could you find something out?

[agronholm]

Process pooling is pretty broken on Python 2.7. I should add something to the docs to strongly discourage its use.

[agronholm]

@srkunze Out of curiosity, would you mind testing on PyPy?

[dalcinl]

@srkunze See python/cpython@dc19c24832f, seems hard to workaround (depending on how much monkeypatchng @agronholm is wiling to accept).

[agronholm]

I can accept patches but first someone needs to explain what is going on and what must be done to fix this. I myself am short on time to delve into this.