macOS: Python-to-Java bridge JPype1 (1.2.0) not working (Java problem)

[RaiMan]

Original problem hint on LaunchPad

from jpype import *
jvm = getDefaultJVMPath()
startJVM(jvm, classpath = '/Users/raimundhocke/IdeaProjects/_SUPPORT/_Latest/2_0_4/sikulixapi-2.0.4.jar')
Screen = JClass('org.sikuli.script.Screen')
s = Screen() # here it hangs trying to get the default Screen object
print(s)

[adrian-evo]

Hi. Just to let you now I published a new SikuliXLibrary for Robot Framework and Python, using JPype instead of Remote Server (the only way to use SikuliX with RF in the past) and so far it works great on Windows: https://github.com/adrian-evo/robotframework-sikulixlibrary

However, on macOS there is this possible issue, but since it also reproduces for simple sample code from JPype and not only for SikuliX (test/test.py from my library), I suspect it is rather a JPype issue than SikuliX. However, looking forward to be able to test my proposed new library also on macOS.

[RaiMan]

@adrian-evo

Excellent work !!! I will have a look into it and will come back, since I am planning, to release a SikuliX version next year, that can be used from Python. JPype is one candidate and py4j the other one.

about the JPype-macOS-SikuliX problem I will dive into the next days. Thanks for your feedback.

[Thrameos]

Does the Screen method access X11? I am fairly sure this is not a JPype bug, but perhaps there is some extra requirement regarding X11 on OSX. We noticed that in some cases adding a matplotlib command before calling certain actions on osx changed behaviors. So it must be something specific to Java on osx.

[RaiMan]

To access the screen I generally use the Java AWT Robot class. I have no idea how this is finally implemented at the system level. But may be it is caused by the new Security&Privacy -> Privacy (Accessibility, Screen Recording) settings, that must be allowed app specific. The problem might be, that the processes are started in the Python environment and the final access is done in the Java environment. Nevertheless I will find a way, to track the situation down to where it hangs.

[RaiMan]

Ok, tracked it down a bit:

• First problem in java.awt.Robot:

    /**
     * Waits until all events currently on the event queue have been processed.
     * @throws  IllegalThreadStateException if called on the AWT event dispatching thread
     */
    public synchronized void waitForIdle() {
        checkNotDispatchThread();
        SunToolkit.flushPendingEvents();
        ((SunToolkit) Toolkit.getDefaultToolkit()).realSync();
    }

Here it hangs (does not come back), which is called internally when completing a mouse action like move.

In the SikuliX class Screen at time of initialization a move to the current mouse location is done, to check whether the mouse is useable. Hence it hangs in something like JClass('org.sikuli.script.Screen')()(trying to create a new Screen object).

The same happens (hangs, does not come back), when trying to dispose an AWT-Window.

• Second problem: does not display an overlay window

... which is based on a special, semi-transparent JFrame (used in Screen.userCapture() and Region.highlight())

The overlay windows are not displayed and it hangs with the first problem.

• when killing a window dispose hang (highlight in this case), I get:

  
  [704 debug] highlight R[774,326 500x500]@S(0) for 2.0 secs

#

A fatal error has been detected by the Java Runtime Environment:

#

SIGSEGV (0xb) at pc=0x000000010f9cf09e, pid=30098, tid=775

#

JRE version: OpenJDK Runtime Environment AdoptOpenJDK (11.0.9+11) (build 11.0.9+11)

Java VM: OpenJDK 64-Bit Server VM AdoptOpenJDK (11.0.9+11, mixed mode, tiered, compressed oops, g1 gc, bsd-amd64)

Problematic frame:

C [_jpype.cpython-39-darwin.so+0xb09e] _ZN7JPClass6invokeER11JPJavaFrameP8_jobjectP7_jclassP10_jmethodIDP6jvalue+0x7e

#

No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

#

An error report file with more information is saved as:

/Users/raimundhocke/PycharmProjects/sx-jpype/hs_err_pid30098.log

Compiled method (nm) 10927 364 n 0 java.lang.Class::isAssignableFrom (native) total in heap [0x0000000120b0c190,0x0000000120b0c548] = 952 relocation [0x0000000120b0c308,0x0000000120b0c338] = 48 main code [0x0000000120b0c340,0x0000000120b0c548] = 520 Disposal was interrupted: java.lang.InterruptedException at java.base/java.lang.Object.wait(Native Method) at java.base/java.lang.Object.wait(Object.java:328) at java.desktop/java.awt.EventQueue.invokeAndWait(EventQueue.java:1361) at java.desktop/java.awt.Window.doDispose(Window.java:1227) at java.desktop/java.awt.Window.dispose(Window.java:1164) at org.sikuli.util.Highlight.close(Highlight.java:269) at org.sikuli.util.Highlight.doShow(Highlight.java:253) at org.sikuli.script.Region.doHighlight(Region.java:2146) at org.sikuli.script.Region.highlight(Region.java:2134) at org.sikuli.script.Region.highlight(Region.java:2112)

This might help the people at JPype.

This is my test script:
``` python
from jpype import *
from jpype import JClass

sx204 = '/Users/raimundhocke/IdeaProjects/_SUPPORT/_Latest/2_0_4/sikulixapi-2.0.4.jar'
sx205 = '/Users/raimundhocke/IdeaProjects/SikuliX1/API/target/sikulixapi-2.0.5-complete-mac.jar'

if __name__ == '__main__':
    jvm = getDefaultJVMPath()
    sxdebug = "-Dsikuli.Debug=4"
    startJVM(jvm, sxdebug, classpath = sx205)

    # to initialize the SikuliX logging
    RunTime = JClass('org.sikuli.script.support.RunTime')
    rt = RunTime.get()

    # the mouse test in class Screen is deactivated (dev version 2.0.5)
    Screen = JClass('org.sikuli.script.Screen')
    s = Screen() # works
    print(Screen, s)
    s.capture() # works

    r = s.getCenter().grow(500) # works
    # s.hover(r.getBottomRight()) # this hangs

    r = s.getCenter().grow(500) # works
    r.highlight(2) # not displayed and hangs
    print(r)

[RaiMan]

@adrian-evo Since I tracked down the problem deep down in the Java AWT area (Robot, Event-Queue), I cannot (and will) not do anything.

For now, I will ignore JPype until this problem is fixed somehow.

[adrian-evo]

@RaiMan Thanks for your effort investigating this. Currently I am not affected since for my work I only need Windows environment and under Windows the JPype with SikuliX works great and it is really fast. macOS might be interesting for other users if they wish to adopt this technology, so I am looking forward for a fix on JPype project side, so that to update my proposed Robot Framework library with the findings.

[Wildprogrammer]

Hi.I encountered a problem, I don’t know if it is similar to this problem？ Sikuliapi：2.0.5-mac.jar macOS catalina:10.15.7 jdk:AdoptOpenJDK 16 my test script:

import os
from jnius import autoclass
os.environ["CLASSPATH"] = r'/Users/design/PycharmProjects/Common/utils/Sikulixapi-2.0.5-mac.jar'
Screen = autoclass("org.sikuli.script.Screen")  
s = Screen()# code will get stuck here

I tried using pyjnius and jpype.They will all get stuck when instantiating the Screen() object.But I can use the function of 'Sikuliide' normally.I have also tried using other sikuli versions (e.g. Sikuli2.0.4.jar).It can pass the above steps, but the function is not normal.E.g. I can find the icon, but I cannot double-click the icon

[RaiMan]

@Wildprogrammer

might be a problem with autoclass.

try the approach from my test script above.

[Wildprogrammer]

@RaiMan Thank you for your help. I tried this code, and they will all get stuck.I also tried the third python library(robotframework-sikulixlibrary).Can be used normally. So do you think it will be the problem with these two python libraries?

[Thrameos]

My general understanding of this issue is that osx has interactions between threads leading to a deadlock between the main and gui threads. This does not occur on linux or windows because they place no restriction on the threading and servicing the GUI. Unfortunately I have no access to an osx machine so I have never been able to replicate it.

Assuming that it can be replicated, the solution I would try is to change the architecture in jpype/native/common/jp_context.cpp such that the the start up creates a separate thread for launching the JVM and the shutdown declares the python main thread as a daemon thread (which makes it so that the Python thread is not longer a hold up for shutdown) and then passes a message to the Java main thread so that the real shutdown can be called. Current JPype and pyjnius are both structured such that the Java and Python share the same main thread which is supposed to result in a deadlock on osx. There was a work around in jpype/_gui.py

def setupGuiEnvironment(cb):
    if _sys.platform == 'darwin':
        from PyObjCTools import AppHelper
        m = {'run': cb}
        proxy = _jproxy.JProxy('java.lang.Runnable', m)
        cbthread = _jclass.JClass("java.lang.Thread")(proxy)
        cbthread.start()
        AppHelper.runConsoleEventLoop()
    else:
        cb()

Unfortunately there is no documentation of how the workaround around was supposed to be used so I can't give any guidance. My guess is that in this a thread which likely is the main application is launched from within Java (which make it say the Python main is now a sub process, and the actual Python main is now transferred to running the AppHelper event loop. I don't really like the structure of this fix as for use in interactive shells and such you really want the Python main thread to be free to interact and have the JVM and AppHelper serviced by the separate worker threads.

Alternative jpype/_core.py could be changed create the threads and manage them within the Python (rather than C++). But again until I can replicate the deadlock and be able to resolve the resulting shutdown issues, I am going to be stuck trying to give guidance to others.

Both Python and Java have the magical concept of a main thread. The main thread is the one that services system calls and such. In Java the main thread is the one that was used to call the startup routine and is the only thread that can call a shutdown (without creating a deadlock) as the shutdown waits for all non daemon threads to terminate before proceeding.