How to handle Jep interpreter when switch between different JAVA thread

[sunyu90325]

Hi, I'm using the newest Jep. I'm on Python 3.9 and Java OpenJ9. I have a question about how to handle multi-threading with Jep. I'm using a sub-interpreter to do some method calls and get the results back in JAVA. But I don't know when to close the interpreter because the JAVA thread is closed automatically. Based on the requirement of Jep, I have to use a different interpreter when I'm calling Python from a different JAVA thread. Since I can't close the previous interpreter from another thread, the previous one is not closed. Without properly closing the previous interpreter, I can initialize a new interpreter but the process is terminated due to an error when I execute "interp.eval(PYTHON_FILE);". And there is no stack trace.

            Long currentTID = Thread.currentThread().getId();
    if(oldTID!=null && !currentTID.equals(oldTID)) {
        interp = new SubInterpreter(config);
                interp.eval(PYTHON_FILE);
    }

Do you know a good way to handle this situation? Thanks!

[ndjensen]

Is there a reason you can't use try-with-resources and your SubInterpreter so it closes automatically when the thread completes?

try (Interpreter interp = new SubInterprereter(config)) {
    // example code here

}  // close automatically called here

Interpreter extends AutoCloseable.

[sunyu90325]

Yes. The subInterpreter does not close properly with try-with-resources. I guess it's due to the use of some Python libraries. SharedInterpreter can close properly but I don't see a way to read in Python file like something below: JepConfig config = new JepConfig(); config.addIncludePaths(PYTHON_PATH); double ret = 0; try (Interpreter interp = new SubInterpreter(config)) { interp.eval(PYTHON_FILE); interp.set("x", x); interp.exec("result = MyPython.calculate(x)"); ret = (double)interp1.getValue("result"); } Now my question is:

1. Do you know other ways to read in Python with SharedInterpreter? If I have to do "interp.exec("PYTHON_CODE");", some of my method is very long. Is there a good way to handle that?
2. I want to visit a Python file with many methods many times within one thread. It could be inefficient to create a Jep instance(a new thread) for each call. Is there a good way to solve that problem? Thanks for your help!

[ndjensen]

I don't understand why SharedInterpreter won't work. You should be able to set the include paths on the JepConfig before the very first SharedInterpreter is created. Alternatively, you can just do something like interp.eval("sys.path.append(newPath")); The include paths are just being added to sys.path. Now if you want different include paths at different times, then yeah, SharedInterpreter won't work.

If the SubInterpreter isn't closing correctly, did you try adding some of your CPython extension modules to the sharedModules? That would prevent the provided modules from being disposed.

In answer to your questions,

1. There is the method Interpreter.runScript(String path).
2. You could keep the thread around by putting a while loop inside the run() method, and having it check a queue for new tasks to run. Then other threads add tasks to the queue.

[sunyu90325]

I said SharedInterpreter can't read in Python file because there is no API available to set JepConfig. The alternative way(interp.eval("sys.path.append(newPath"))) you provided works great for me. Thanks so much to help me out! I thought about keeping a standalone thread running for Jep. However, it won't be thread-safe. For example, if I use interpreter.set("x", x) in one thread, and there is another thread doing the same thing, the result may not be correct. I have to create a Jep instance and read in Python the file there for each call, right?

[ndjensen]

You can use a JepConfig on a SharedInterpreter if you don't mind it applying to all SharedInterpreter instances. See https://ninia.github.io/jep/javadoc/4.0/jep/SharedInterpreter.html#setConfig-jep.JepConfig-

[sunyu90325]

That's strange. I use Jep 4.0.1. The compiler doesn't allow me to use the setConfig method for SharedInterpreter.