Make interactive mode work on macOS

elevans commented 8 months ago

Currently, macOS users are unable to use interactive mode with PyImageJ. See issue https://github.com/imagej/pyimagej/issues/23 for more details on getting the ImageJ GUI to work on macOS. Work on this issue resulted in the gui mode for PyImageJ macOS users. While the GUI does work for macOS users, they unfortunately loose access to the REPL (unlike Linux and Windows users). This is because on macOS the Java AWT GUI must poll the interface for clicks and other input using Apple's AppKit framework (accessible via PyObcTools from Python). This is done via an NSRunLoop which blocks the console. In Python land this is done with AppHelper.runConsoleEventLoop() from pyobjc.

Here is a quick breakdown of my conclusions thus far in investigating this issue:

I have tried a variety of Python threading strategies at all relevant levels of the PyImageJ stack: pyimagej, scijava and jpype. Trying to start and control the JVM and AppHelper event loop from Python doesn't work.
I think (as well as some others in posts around the web) the AppHelper.runConsoleEventLoop() and the associated Objective-C code checks if its in the main thread or not. This means that any viable approaches here will likely involve sacrificing the main thread to Apple.
The concurrent.futures module is just a high level API for the threading module. It doesn't offer any solutions here.
Using the installInterrupt=True flag with AppHelper.runConsoleEventLoop() doesn't help here.
Using Python's asyncio doesn't work (I also tried with installInterrupt=True).
Using the multiprocessing module and starting a new process works...but of course the two processes can't share data easily. You can make a shared memory map and share data but this seems like it would require extensive rework. I didn't try this. Also, appose could make this work nicely for us as well.

The latest approach I've seen and have tried (it didn't work but I could have been doing it wrong) is using AppKit's dispatch mechanism. The credit for this idea goes to this stackoverflow post where the user retsigam wanted to interact with the Bluetooth stack (which needed to run a console event loop) without locking up the REPL. This seemed like as close as I was going to get to finding an analogous problem in the wild. This solution relies on the libdispatch which is also known as Grand Central Dispatch.

Digging a little further pyobjc added bindings for libdispatch in version 4.1. See this issue. Also in that issue is another user who is trying to resolve the console event loop block by starting the AppHelper.runCopnsoleEventLoop() in a separate thread. Wayne solved this by using the dispatch mechanism documented here. This made think that using the dispatch mechanism was a viable approach here.

NicoKiaru commented 8 months ago

I'm not bringing any useful comment, but thanks a lot for all the digging @elevans and @ctrueden. This looks pretty painful.

oeway commented 6 months ago

Any news on this issue? We'd like to create a LLM agent for pyimagej, but this prevent me from having both a chatbot and imagej interface.

ctrueden commented 6 months ago

Yes! Just yesterday I completed my draft implementation of Python support for Jaunch, the native launcher I've been working on to replace the current ImageJ Launcher. If you launch Python via Jaunch instead of the normal python binary, it spins up the Python interpreter off the main thread when using macOS, such that you can then import imagej; ij = imagej.init(mode='interactive'); ij.ui.showUI(), and it works! :tada:

We still need to update PyImageJ to remove the "no interactive mode on macOS" fail-fast check, though. And of course, Jaunch is not yet released, so all this is quite "bleeding edge" for the moment. But there is hope.

@oeway Do you think launching via Jaunch would be sufficient for you? Or do you need everything work via the standard python tool? If you need it to work via python, what about the GUI mode? This works now to show an ImageJ GUI; it just hangs up the process's main thread with the AppKit loop, replacing the REPL. (Normally the Python REPL runs on the main thread.)

One other idea to make interactive mode work from standard python: write a Cython program that calls Py_BytesMain on a separate pthread with the script:

import code; code.interact(local=locals())

or

import IPython
IPython.embed()

and then in the main Python program, start the AppKit loop. So the main program's thread becomes blocked, but meanwhile there is a Python REPL running on another thread. I haven't tried it yet though, so it may or may not work...

oeway commented 5 months ago

Hi, @ctrueden This sounds great! Great that you made it work with Jaunch. Please let me know how I can try it.

I am trying to make this hypha service for imagej work: https://gist.github.com/oeway/4fb456ab1d700e802d76a98e88370359 (the same one as we worked during our last hackathon in loci). This script launches imagej and expose some python function to a remote hypha server (similar to imagej http server plugin). Here we specifically want the user to be able to see the GUI and operate together with the remote request. But I could not get it to work with mac. I just tried GUI mode and it crashes:

Testing the imagej service...
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007ff81d1967db, pid=48204, tid=5647
#
# JRE version: OpenJDK Runtime Environment (14.0.1+14) (build 14.0.1+14)
# Java VM: OpenJDK 64-Bit Server VM (14.0.1+14, mixed mode, sharing, tiered, compressed oops, g1 gc, bsd-amd64)
# Problematic frame:
# C  [libobjc.A.dylib+0xe7db]  class_getSuperclass+0x5
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /Users/wei.ouyang/workspace/bioimageio-chatbot/hs_err_pid48204.log
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
zsh: abort      python scripts/start_imagej.py

If I set to headless, it looks like running, but still got stuck somehow. Maybe related to my JVM? No idea.

I would be happy to try Jaunch, could you let me know how I can get the imagej gui together with this python script?

imagej / pyimagej

Make interactive mode work on macOS #298