search

ericsnowcurrently / multi-core-python

Enabling CPython multi-core parallelism via subinterpreters.
BSD 3-Clause "New" or "Revised" License
245 stars 6 forks source link

concurrent.futures.InterpreterPoolExecutor #45

Closed crusaderky closed 5 years ago

crusaderky commented 5 years ago

As one of the logical consequences to PEP 554, it would be neat to have a concurrent.futures.InterpreterPoolExecutor.

I wrote the initial code at https://github.com/crusaderky/subinterpreters_tests - currently missing unit tests and pickle5 buffers support.

If everybody is happy with the design, I'll start working on a pull request to CPython as soon as https://github.com/ericsnowcurrently/multi-core-python/issues/34 is resolved (it's kinda pointless before that).

ericsnowcurrently commented 5 years ago

Whoa, that's really cool! :) I have talked briefly with Davin Potts about integrating subinterpreters into multiprocessing, but hadn't thought about concurrent.futures. So thanks for looking into this.

I recommend opening an issue on bugs.python.org for this as soon as it makes sense. This repo is unofficial, effectively just a planning tool with a specific goal in mind, and mostly just me. :) The official BPO tracker is where the real collaboration happens. So I'd hate for your work to not get enough attention.

FWIW, I am keeping track of a few things that I hope to work on after PEP 554 is landed, but mostly everything here is targeting the minimal functionality to hit the core goal. :)

ericsnowcurrently commented 5 years ago

https://bugs.python.org/issue37293

@crusaderky, thanks!