OPT: add check before submit incase of memory exhausted;

[NoneGG]

problem

I found memory exhausted when i use concurrent.futures.ProcessPoolExecutor in prod env. The code is similar to code below:

#!/usr/bin/python
# -*- coding: utf-8 -*-
import concurrent.futures
import time

__author__ = 'chenming@bilibili.com'

def worker(array):
    # io takes time to finish
    time.sleep(10)

def main():
    with concurrent.futures.ProcessPoolExecutor(max_workers=4) as executor:
        for i in range(10):
            # a normal size dict, will be copied to subprocess but should be free when the subprocess finish its work
            executor.submit(worker, {k: 1 for k in range(1000000)})
        time.sleep(10)

if __name__ == '__main__':
    main()

reason

I found that if subprocess takes longer time to finish than submit (which is almost determined), the ProcessPoolExecutor._pending_work_items will be full of param submitted, and the dict seems to be too large, then the memory is exhausted. By the way, i doubt if del in Python2 can really release memory to system (seems to be optimized in Python3).

how to avoid

I add a check of _pending_work_items in ProcessPoolExecutor.submit before actually add new workItem to _pending_work_items. If len(self._pending_work_items) > self._max_workers + EXTRA_QUEUED_CALLS, it will block and wait for workItems in _pending_work_items be deleted.

[agronholm]

Does upstream code have this? If not, I suggest you take it up with them first.

[dalcinl]

If this behavior is ever proposed upstream, my bet is that core Python developers will reject it in limine.

• The whole point of an asynchronous framework like concurrent.futures is to never block.
• The patch proposed in this PR is unacceptable, you cannot just naively busy-loop waiting for item in the pending work to be consumed, that's not the way we do multithreading in Python!
• If memory pressure is an issue, this dirty fix proposed here is rather trivial apply by either subclassing or monkeypatching. But an asynchronous framework should not trade memory pressure for blocking behavior.

@agronholm I would reject and close this PR.

[agronholm]

The whole point of an asynchronous framework like concurrent.futures is to never block.

concurrent.futures is not an asynchronous framework. Future.result() already blocks, by design of course. I do concur with the rest of the comments though.

[dalcinl]

@agronholm Sorry for my bad wording, I meant asynchronous execution framework, and the never block of course does not apply to Future.result().

[NoneGG]

Sure, thank you for your feedback~