search

JuliaPy / PythonCall.jl

Python and Julia in harmony.
https://juliapy.github.io/PythonCall.jl/stable/
MIT License
776 stars 63 forks source link

Support for Multi-Threading with PythonCall? #330

Closed nic-barbara closed 1 year ago

nic-barbara commented 1 year ago

Are there any plans to support multi-threading with PythonCall.jl in the near future? Almost everything about this package is superior to PyCall.jl except for thread safety.

Specifically, it would be great (and in my case - is essential) to be able to call python functions on many threads in parallel. This is useful when a core routine is written in python and needs to be called many time independently.

For example, some environments in ReinforcementLearning.jl rely on the python bindings to MuJoCo. Evaluating these environments efficiently requires multi-threading (eg: this implementation with PyCall.jl), which is currently impossible in PythonCall.

Is this something you would consider supporting in the near future?

github-actions[bot] commented 1 year ago

This issue has been marked as stale because it has been open for 30 days with no activity. If the issue is still relevant then please leave a comment, or else it will be closed in 7 days.

cjdoris commented 1 year ago

Because of the GIL, Python itself has poor support for multithreading, because only one thread can hold the GIL at a time, and this is required to do anything Pythonic. Hence multithreading is only useful in Python if you are doing a lot of work which can release the GIL (such as IO or long-running numerical routines written in C).

In general, if you want to parallelise Python it's generally better to use multiprocessing instead of multithreading. So there are no plans for PythonCall to support it.

github-actions[bot] commented 1 year ago

Thank you for taking the time to report this issue! However, it has been marked as stale because there has been no activity for 30 days. As with many open source projects, the maintainers work for free and do not always have enough time to address all issues. For questions, please check the documentation, other issues, or consider asking elsewhere such as Stack Overflow or Julia Discourse. For bugs, please try to find a fix yourself and consider opening a pull request. If you're still stuck, please leave a comment below. If there is no further activity in 7 days, this issue will be automatically closed.