Calling Python from Chapel; any interest?

[astatide]

As part of some work I was doing, I explored the possibility of calling Python functions/modules from within Chapel. This was motivated by the desire to create a software package that was scriptable without needing an end user to learn Chapel/recompile the binary.

I used the C api to embed the CPython interpreter, then wrote C functions that called, and returned, whatever python function I wanted. There were some difficulties regarding threading that were fairly easy to solve (for many standard use cases), but otherwise handling the Python data types was simple; CPython includes a lot of macros that will unpack the structs for you. The code was both an embedded interpreter and a python module, so I could also expose data from the Chapel side in a trivial manner (for instance, exposing non distributed arrays as numpy modules).

I was able to successfully run parallel numpy operations by calling the function from Chapel and achieved the standard performance you'd expect from Python modules that leverage C code (in this case, 'perfect' scaling).

I'm curious if there's any interest in this, both from a developer support and end user perspective. I can imagine such a thing being useful where recompilation is impossible or undesirable, or where you want to make algorithm changes on the fly. All in all, it was probably only 100 lines of code. There are many edge cases I never tested (and obviously I had to do some work to support arbitrary data types) but.

[bradcray]

I think that being able to call from Chapel to Python would definitely be attractive.

[astatide]

I can clean up the existing code and provide a module for someone to test, if they're interested, in that case.

[ben-albrecht]

@astatide - I would suggest putting together a mason package to share the work you have done so far and to get feedback.