search

ericsnowcurrently / multi-core-python

Enabling CPython multi-core parallelism via subinterpreters.
BSD 3-Clause "New" or "Revised" License
245 stars 6 forks source link

Get Cython to produce subinterpreter-compatible extensions. #54

Open ericsnowcurrently opened 5 years ago

ericsnowcurrently commented 5 years ago

At the Python core sprint this year I asked @scoder about Cython support for subinterpreters. He explained that Cython implements PEP 489 (multi-phase init) but does not support subinterpreters (even though implementing PEP 489 implies support :disappointed:). In particularly Cython generates extensions with non-const global variables (e.g. statics). Anything we can do to help Cython support subinterpreters will go a long way to helping extension authors.

encukou commented 5 years ago

Full support for multiple subinterpreters would of course be amazing, but I'd like to be clear that it'll be an enhancement. Today, Cython is doing nothing wrong.

Cython's extensions can only be imported in one interpreter. An import in another interpreter fails cleanly. The meaning of "to support subinterpreters" has changed somewhat in the last five years, but we (the PEP's authors) did mean Cython's current behavior to count as implementing multi-phase init correctly.

ericsnowcurrently commented 5 years ago

Yeah, you're right. I should have been more clear.

eduardo-elizondo commented 5 years ago

@ericsnowcurrently We are already eyeing this! It indeed possible to achieve it by changing the code generator to follow the patterns that we have been introducing in CPython's C Extensions. That being said, I would really like to get a critical mass in CPython before getting to any external project.

If we get more changes merged, I can easily get to the Cython generator. Currently, I'm just blocked on reviews - there are 3 pending PR and I don't want to over saturate the CPython's PRs 😄

On that note, I'm almost done with an updated C Extension tutorial that does not use any statics. So expect that soon!