search

ninia / jep

Embed Python in Java
Other
1.3k stars 147 forks source link

Does this library support setup of multiple interpreters configured with different python versions? #434

Closed jradlica closed 1 year ago

jradlica commented 1 year ago

Describe the problem Does this library support setup of multiple interpreters configured with different python versions? I've seen https://github.com/ninia/jep/issues/76 but as far as I've understood the hints described in this issue shows how to run all interpreters on single virtualenv. (I have need of running multiple interpreters on multiple virtualenvs)

Let's assume that I need to have two interpreters in my app:

Is there any option to configure interpreters as above? (I need the confirmation) Are there any plans for adding such support?

Thank you in advance for your answers.

bsteffensmeier commented 1 year ago

Is there any option to configure interpreters as above? (I need the confirmation)

No

Are there any plans for adding such support?

No

jradlica commented 1 year ago

@bsteffensmeier Thank you for fast reply!

lfcnassif commented 11 months ago

Just to let the project know we hit this libraries conflict situation. It would be very nice to think about including such support in possible future plans. Thanks for this great project.

PS: Actually our use case is not loading different python versions, but different and incompatible transitive dependency versions used by our direct python dependencies.

ndjensen commented 11 months ago

I don't see how it would be possible to support different versions of Python at the same time. libjep is compiled and linked against libpython and it won't work if it's not matching the version it was compiled and linked against. Also, the Jep native code uses more functions than those provided by the CPython stable ABI, so that would be an issue when switching versions. Even if we did only use the stable ABI, if you wanted to just switch at runtime, you'd have to somehow unlink and relink to the new version of libpython.

@lfcnassif, if your dependencies are written purely in Python, then sub-interpreters should work and properly isolate the dependencies. However, if some of the dependencies are CPython extension modules, i.e. have some C code, then those most likely won't work well with sub-interpreters, and that's on the extension module's maintainers to fix, not Jep. Even if they wrote it to work well with sub-interpreters, you could only have the one version of the library loaded at a time in the process, and would have to go through the hoops of trying to get it unloaded and then reloaded when switching.

lfcnassif commented 11 months ago

I don't see how it would be possible to support different versions of Python at the same time.

I'm sorry @ndjensen, actually my need would be to use the same python version, but to load different library versions depending on the job, as I updated in my PS above.

@lfcnassif, if your dependencies are written purely in Python, then sub-interpreters should work and properly isolate the dependencies. However, if some of the dependencies are CPython extension modules, i.e. have some C code, then those most likely won't work well with sub-interpreters, and that's on the extension module's maintainers to fix, not Jep. Even if they wrote it to work well with sub-interpreters, you could only have the one version of the library loaded at a time in the process, and would have to go through the hoops of trying to get it unloaded and then reloaded when switching.

Unfortunately we have to deal with native dependencies (tensorflow, pytorch, dlib). But I'll try to accomplish our goal using sub-interpreters, thanks for the advice!

Thank you very much @ndjensen for all your great explanations and for your time!