Problem with executing several independent Python scripts

[Daniel-Alievsky]

Hello! Could you advise me?

I want to use Jep in a multi-user desktop/server Java-based system, where even one user may open a set of different solutions (projects) in different windows (server sessions). In every solution, a user may edit and run any number of different scripts, written in JavaScript and (now) in Python.

In JavaScript (GraalVM) it is not a problem: I can create a separate ScriptEngine for every project or even for every separate script. Different engines are isolated from each other and do not interfere with each other.

But in Jep all is not so simple. If I create instances of SharedInterpreter, they will interfere. For example, changing sys.path will affect ALL projects, executing in the same JVM now (right?) And I cannot avoid correcting sys.path, as well as user's Python scripts may use some custom .py-files, placed in a directory, specific for this project. I don't see a good solution of this problem. Of course, I can add many paths to sys.path, but what to do if two projects contain some script "demo/VerySimple.py" with the same name? Moreover, a user may close a project and remove (or move) its directory to some other place - what will be with all other opened projects? It seems the only way is to strongly restrict a folder for placing any custom Python modules on a single computer...

Of course, I also tried SubInterpreter. But it prints:

UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. import numpy as np

Moreover, really, when I tried to use two instance of SubInterpreter in the same thread, the second one lead to system crash: EXCEPTION_ACCESS_VIOLATION (I can sent you a link to my experimental GIT if you want to reproduce this.)

Can you advise me anything?

[bsteffensmeier]

Are you including numpy in the list of shared modules when you create your sub-interpreter? This will cause numpy to be shared between sub-interpreters which can prevent it from crashing while still allowing other things, like sys.path, to remain isolated.

[Daniel-Alievsky]

But I'm going to use a lot of other native libraries, from OpenCV to neural networks. As I understand, it is the common problem? If I want to use Jep with external libraries, then I can be safe only if I create 1 SharedInterpreter per 1 JVM. Right?

And, on the other hand, if I want to use Python only as a pure script without native libraries, like JavaScript, I may freely create a lot of isolated "Python machines" by separate SubInterpreter instances?

[bsteffensmeier]

But I'm going to use a lot of other native libraries, from OpenCV to neural networks. As I understand, it is the common problem?

Yes this is a common problem, This is why we have introduced a concept of Shared Modules and Shared Interpreters. Most users are able to get jep working with their application using one of these two approaches. If shared interpreters are not an option for you, it is often possible to workaround the problems with sub interpreters by including numpy and other incompatible libraries in the list of shared modules in the JepConfig.

If I want to use Jep with external libraries, then I can be safe only if I create 1 SharedInterpreter per 1 JVM. Right?

You can create more than one SharedInterpreter but they must be created on separate threads. They will share some state as you mentioned above.

And, on the other hand, if I want to use Python only as a pure script without native libraries, like JavaScript, I may freely create a lot of isolated "Python machines" by separate SubInterpreter instances?

It is possible to write a native library that can be safely used in sub-interpreters, that is a choice that is up to the maintainer of the libraries you are using and is outside of the control of jep. There is more information about how native libraries can interact with sub-interpreters in PEP-554, PEP-630, and PEP-3121.

[Daniel-Alievsky]

Thank you for explanations!