ninia / jep

Embed Python in Java
Other
1.27k stars 146 forks source link

Problem with initialising SharedInterpreter twice causing pthread_mutex_lock+0x4 #523

Open Loc300 opened 3 months ago

Loc300 commented 3 months ago

I´m using Jep in my Java Gradle project and use it to predict with pytorch neural network.

My initialising looking like that

public class Test { private SharedInterpreter interp = new SharedInterpreter(); public Test() throws IOException { interp.eval("import sys"); interp.eval("sys.path.append('')"); interp.runScript("/Users/test.py"); } } When I initialize the class for the very first time and execute my prediction, everything works so far. The line for the prediction is pretty much Object result = interp.invoke("predict_test", params);

So i can run my predict and train functions as often as i like, for that first SharedInterpreter instance. After i´m done with the training is use interp.close(). To avoid the error here:

Exception in thread "main" jep.JepException: JEP THREAD WARNING: Unsafe reuse of thread main for another Python Interpreter. Please close() the previous Interpreter to ensure stability.

So in my case after one succesful training, im creating a new instance of the Testclass. And when im using my prediction function again, im getting the following error:

A fatal error has been detected by the Java Runtime Environment: SIGSEGV (0xb) at pc=0x00007ff815e1f3e4, pid=3753, tid=0x0000000000002803 JRE version: OpenJDK Runtime Environment (8.0_302-b08) (build 1.8.0_302-b08) Java VM: OpenJDK 64-Bit Server VM (25.302-b08 mixed mode bsd-amd64 compressed oops) Problematic frame: C [libsystem_pthread.dylib+0x13e4] pthread_mutex_lock+0x4 somtimes: C [Python+0x12d713] take_gil+0x30

When i dont create a new instance of the Testclass, the functions work very well.

Am i missing something?

Environment (please complete the following information): Im Using an M1 Macbook Pro

Sonoma 14.4 (23E214) Python 3.11 JEP==4.2.0 Python packages are mostly pytorch, numpy, logging and json

bsteffensmeier commented 3 months ago

This is not the expected behavior but it can be very difficult to identify whether this is a problem with jep or with one of the libraries you are using. If possible I recommend running your test without any imports and verify jep works with the base python interpreter. If that works then it is likely this isn't a problem with jep but rather a problem with one of the libraries. If a library holds state information about an interpreter in native memory then it may not be possible to use the library in multiple interpreters, In this case you may need to maintain a single SharedInterpreter for use whenever you access the library.