search

python / cpython

The Python programming language
https://www.python.org
Other
62.37k stars 29.95k forks source link

concurrent.futures.InterpreterPoolExecutor #81474

Open 1c23ebe2-087a-4be2-9b19-c5cb5aeb1577 opened 5 years ago

1c23ebe2-087a-4be2-9b19-c5cb5aeb1577 commented 5 years ago
BPO 37293
Nosy @ericsnowcurrently, @applio, @crusaderky, @jakirkham

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['extension-modules', 'type-feature', '3.9'] title = 'concurrent.futures.InterpreterPoolExecutor' updated_at = user = 'https://github.com/crusaderky' ``` bugs.python.org fields: ```python activity = actor = 'jakirkham' assignee = 'none' closed = False closed_date = None closer = None components = ['Extension Modules'] creation = creator = 'crusaderky' dependencies = [] files = [] hgrepos = [] issue_num = 37293 keywords = [] message_count = 3.0 messages = ['345681', '346225', '346226'] nosy_count = 4.0 nosy_names = ['eric.snow', 'davin', 'crusaderky', 'jakirkham'] pr_nums = [] priority = 'normal' resolution = None stage = 'needs patch' status = 'open' superseder = None type = 'enhancement' url = 'https://bugs.python.org/issue37293' versions = ['Python 3.9'] ```

1c23ebe2-087a-4be2-9b19-c5cb5aeb1577 commented 5 years ago

As one of the logical consequences to PEP-554, it would be neat to have a concurrent.futures.InterpreterPoolExecutor.

I wrote the initial code at https://github.com/crusaderky/subinterpreters_tests - currently missing unit tests and pickle5 buffers support.

If everybody is happy with the design, I'll start working on a PR as soon as the GIL becomes per-interpreter (it's kinda pointless before that).

ericsnowcurrently commented 5 years ago

FWIW, performance benefits when subinterpreters stop sharing the GIL are not the only benefit. In fact, PEP-554 is specifically written to avoid that consideration, focusing on the benefits of the concurrency model (i.e. CSP). So I wouldn't call this pointless even without per-interpreter GIL. :)

(Of course, this *would* be blocked on acceptance and implementation of PEP-554.)

ericsnowcurrently commented 5 years ago

@Davin, we've spoken before about something similar for multiprocessing, IIRC. :)