MainInterpreter hangs on initialize()

[dlnorgaard]

Describe the problem I am using Jep in Scala and have been able to successfully run Python code using JEP in unit tests. When I go to run our main application though, it is unable to initialize neither shared nor sub interpreters. It doesn't return any errors but instead hangs on instance.initialize() in MainInterpreter.getMainInterpreter(). It looks like it tries to start the thread and waits for it, but doesn't get there. For JepConfig, I do addIncludePaths to add the location of site-packages in our python virtual environment using absolute path. I have been careful to close the interpreter after each use, though currently there is only one place outside unit tests that is opening a sharedinterpreter. If you have any ideas as to what might be going wrong, please let me know.

Environment (please complete the following information):

• OS Platform, Distribution, and Version: Windows 10
• Python Distribution and Version: Python 3.8
• Java Distribution and Version: JDK 1.8.0 update 331
• Jep Version: 4.0.3
• Python packages used (e.g. numpy, pandas, tensorflow): miniKanren, local

[dlnorgaard]

It hangs on initializePython here:

    thread = new Thread("JepMainInterpreter") {

        @Override
        public void run() {
            try {
                initializePython(sharedModulesArgv);
            } catch (Throwable t) {
                error = t;
            } finally {
                synchronized (MainInterpreter.this) {
                    MainInterpreter.this.notify();
                }
            }

I'm not able to step into it.

[dlnorgaard]

I should mention I am running all of this in IntelliJ 2022.1.

[ndjensen]

It sounds like you're doing everything correctly. Also we may have seen this before but it's unsolved, it sounds like #352. See my last comment there for some ideas.

A couple things to check. Since you're on Windows, did you get a jep.bat script and does that work if you run it? By unit tests, were you referring to the jep unit tests launched by running python setup.py test or your own unit tests? Either way, if those are working, try checking the environment variables (PATH, PYTHONHOME, anything else Python related) of your hanging application vs the tests.

Last but not least, if you're sure it's in the native C code that's hanging, you could try like I recommended on #352 and add some printfs to pyembed_startup() to try and find the exact line that's hanging and then we could go from there.

[dlnorgaard]

It does sound like the same issue.

I did not get a jep.bat script. The unit tests were my own to make sure I can do what I want to do with JEP from Scala. Our main app takes a long time to load so it's just more efficient to work out kinks that way.

A colleague of mine got his working somehow, after initially having the same problem. The main things he did was ensuring interpreters are closed. I can run his code, but mine doing nearly the same thing, does not work. So far I haven't had much luck figuring out the difference but will look again. I'll also try your suggestion about the printf statements when I get a chance . Will report back with any findings.

[dlnorgaard]

Added print statements, rebuilt, and installed that version. Would I expect to see the print statements in console? Not seeing anything. I put something at the very beginning, as well as throughout the function:

void pyembed_startup(JNIEnv *env, jobjectArray sharedModulesArgv) { printf("very beginning");

Just a heads up that I'm no C programmer....

[dlnorgaard]

Well, what I found out after looking at my colleague's code again, is that he had a dangling interpreter that doesn't get closed, e.g.:

val jepConfig: JepConfig = new JepConfig()
jepConfig.setIncludePath(venv.getAbsolutePath)
SharedInterpreter.setConfig(jepConfig)
val danglingInterpreter = new SharedInterpreter()

For some reason this allows other interpreters to initialize without hanging.

[ndjensen]

Add a \n to your printf, so printf("very beginning\n"). You might need the \n to flush the output, not sure. Yes the output should show in the console if it gets that far.

I don't think the colleague's dangling interpreter is what makes it work for him. We already dangle the MainInterpreter that initializes Python, and we only initialize Python once, so he has already gotten past the point where you see a hang.

Do you have more than one Python installed on your system?

[dlnorgaard]

Adding \n didn't help. I added the dangling interpreter to my code and that got it working, so I think it does do something. I only have one Python installed, although I am using a virtual environment which I add to JepConfig. Problem seems to be the same with or without setting JepConfig.

[ndjensen]

I don't understand your problem. If you have a dangling interpreter to keep around, then you successfully initialized MainInterpreter and Python, and did not hang at those steps.

[dlnorgaard]

Probably I am misunderstanding something, but I thought dangling interpreters were a bad idea and should be closed. So if I do something like below, initializing the interpreter hangs in our main app (though it works in test code). The below works if I have that dangling interpreter in addition. Seems like a strange behavior. Btw, I never got any of the print statements out on console, even when it's working.

try {
  val interpreter = new SharedInterpreter()
  try {
    interpreter.runScript(script)
    interpreter.exec("s = dosomething")
    etc
  } catch {
    case e: Exception =>
      throw e
  } finally if (interpreter != null) interpreter.close()
} catch {
  case e: Exception =>
    throw e
}

[ndjensen]

When you create an Interpreter instance, one of the first things that happens is the MainInterpreter is initialized, and only initialized once. So if it's hanging, I don't understand how you managed to create two separate Interpreter instances in your example. It either hung the first time or didn't hang, does that make sense? I don't understand how you got in a state where you were able to create a second Interpreter instance if the MainInterpreter is hanging.

Yes, ideally you should not have dangling interpreters, though we dangle the MainInterpreter for stability.

If you didn't get the prints, were you building from source and then installing with python setup.py install while the virtualenv was activated? If not, it may be picking up a different Jep that was installed and not replaced.

I installed both via pip and via source on Windows and got a jep.bat script in the virtualenv's Scripts directory alongside other executables like python.exe and pip.exe. Please check if you have jep.bat and if so, run it and see if it works and opens an interactive interpreter that can import Java classes.