DYLD_LIBRARY_PATH in environment causes fatal error in Java Runtime Environment on M2

[tdyoshida]

I've been using paquo quite a while (thanks for developing this awesome package!), but now I'm encountering a problem importing paquo.projects. I use m2 Macbook pro

I created a new conda environment and installed paquo as conda create -n py_paquo -c conda-forge paquo, and the installation was successful. I was able to import paquo without any errors. paquo version is 0.8.0

However, when I try: from paquo.projects import QuPathProject, there is a following error.

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x0000000000000000, pid=6130, tid=259
#
# JRE version:  (17.0.8.1+1) (build )
# Java VM: OpenJDK 64-Bit Server VM (17.0.8.1+1, mixed mode, tiered, compressed class ptrs, g1 gc, bsd-aarch64)
# Problematic frame:
# C  0x0000000000000000
#
# Core dump will be written. Default location: /cores/core.6130

Detailed error report was also created, and I'm attaching it as a separate file. hs_err_pid6130.log

The above error is when using QuPath 0.4.4. I also tried QuPath-0.5.1-arm64 version, and I got a different error by from paquo.projects import QuPathProject.

Error occurred during initialization of VM
Could not resolve "ZIP_Open": /Applications/QuPath-0.5.1-arm64.app/Contents/runtime/Contents/Home/lib/libzip.dylib

Thank you for your help!

[ap--]

Hi @tdyoshida

This usually happens if your python interpreter was build for a different architecture than your QuPath version. I fixed an issue with the paquo get_qupath commandline script that now downloads the correct version of QuPath when you are running on an arm architecture mac. The new release paquo==0.8.1 should be available on pypi and conda-forge soon.

The instructions below were tested on a M2 mac for qupath 0.4.4 and 0.5.1

$ mamba create -n py_paquo -c conda-forge paquo

$ mamba activate py_paquo

$ python -c "import platform; print(platform.machine())"
arm64

$ paquo get_qupath --install-path . 0.4.4
# extracting: /var/folders/l4/gpyzwv455hzbmbv78x05555h0000gn/T/QuPath-0.4.4-Mac-arm64.pkg
# available at: /Users/andreaspoehlmann/test-paquo-arm/QuPath-0.4.4.app
#
# use via environment variable:
#  $ export PAQUO_QUPATH_DIR=/Users/andreaspoehlmann/test-paquo-arm/QuPath-0.4.4.app
#
# use via .paquo.toml config file:
#  qupath_dir="/Users/andreaspoehlmann/test-paquo-arm/QuPath-0.4.4.app"
/Users/andreaspoehlmann/test-paquo-arm/QuPath-0.4.4.app

$ export PAQUO_QUPATH_DIR=/Users/andreaspoehlmann/test-paquo-arm/QuPath-0.4.4.app

$ paquo --qupath-version
0.4.4

$ python -c "import paquo.projects"

Let me know if this solves your issue.

Cheers, Andreas 😃

[tdyoshida]

Thank you for your response!

I installed paquo 0.8.1 and used it with QuPath 0.5.1, but unfortunately it did not solve the problem in my environment. There's basically the same error (below is the paquo version).

# Name                    Version                   Build  Channel
paquo                     0.8.1              pyhd8ed1ab_0    conda-forge

I also tried to follow what you described: Created a new conda environment and installed a separate QuPath. However, paquo get_qupath --install-path . 0.4.4 gave the following error.

# extracting: /var/folders/ws/mlbxk4rj50sgvnxjyp861qqr0000gn/T/QuPath-0.4.4-Mac-arm64.pkg
Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniconda/base/envs/py_paquo/bin/paquo", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/py_paquo/lib/python3.12/site-packages/paquo/__main__.py", line 89, in main
    return args.cmd_func(args)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/py_paquo/lib/python3.12/site-packages/paquo/__main__.py", line 349, in get_qupath
    app = extract_qupath(file, args.install_path, system=system)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/py_paquo/lib/python3.12/site-packages/paquo/_utils.py", line 247, in extract_qupath
    raise RuntimeError("no qupath extracted?")
RuntimeError: no qupath extracted?

As you notice, my conda is miniconda installed via Homebrew (not sure the error has anything to do with it).

When I tried paquo get_qupath --install-path . 0.5.1, it successfully downloaded QuPath. However, after setting PAQUO_QUPATH_DIR (below), paquo --qupath-version gave an error.

$ echo $PAQUO_QUPATH_DIR
/Users/tdyoshida/test_paquo_arm/QuPath-0.5.1.app

$ paquo --qupath-version
Error occurred during initialization of VM
Could not resolve "ZIP_Open": /Users/tdyoshida/test_paquo_arm/QuPath-0.5.1.app/Contents/runtime/Contents/Home/lib/libzip.dylib

$ python -c "import paquo.projects"
Error occurred during initialization of VM
Could not resolve "ZIP_Open": /Users/tdyoshida/test_paquo_arm/QuPath-0.5.1.app/Contents/runtime/Contents/Home/lib/libzip.dylib

Any idea what might be wrong?

Thank you for your help!

[ap--]

Hi @tdyoshida

It's a bit hard to debug. Can you please run the following bash script and post ALL the output here?

#!/bin/bash

set -euxo pipefail

eval "$(conda shell.bash hook)"
mkdir ~/new_paquo_test
cd ~/new_paquo_test
conda create -n paquo_test -c conda-forge paquo
conda activate paquo_test
python -c "import platform; print(platform.machine())"
paquo get_qupath --download-path . --install-path . 0.4.4 | tee .qupath_install_path

export PAQUO_QUPATH_DIR=$(cat .qupath_install_path | grep -v "^#")

paquo --qupath-version
python -c "import paquo.projects"

[tdyoshida]

Hi @ap-- , thank you for your response!

I run the script you provided, and the log is attached (paquo_test.log). It appeared to be completed successfully (no obvious error message in the log).

However, when I manually activated the conda env paquo_test (created by your script), set PAQUO_QUPATH_DIR, and run python -c 'import paquo.projects', there was a following error:

$ echo $PAQUO_QUPATH_DIR
/Users/tdyoshida/new_paquo_test/QuPath-0.4.4.app

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x0000000000000000, pid=13365, tid=259
#
# JRE version:  (17.0.8.1+1) (build )
# Java VM: OpenJDK 64-Bit Server VM (17.0.8.1+1, mixed mode, tiered, compressed class ptrs, g1 gc, bsd-aarch64)
# Problematic frame:
# C  0x0000000000000000
#
# Core dump will be written. Default location: /cores/core.13365
#
# An error report file with more information is saved as:
# /Users/tdyoshida/hs_err_pid13365.log
#
#
[1]    13365 abort      python -c 'import paquo.projects'

I'm also attaching the log from above. paquo_test.log hs_err_pid13365.log

One thing I noticed is that I cannot open the newly downloaded QuPath from Mac GUI. It showed an error that The application "QuPath-0.4.4" can't be opened. Is it an expected behavior?

Thank you so much for your kind help!

[ap--]

Hmmm... So the fact that the script ran without error means that this is not an issue on the paquo side.

Can you try running these commands after each other in your command line:

cd ~/new_paquo_test
conda activate paquo_test
which python
export PAQUO_QUPATH_DIR=/Users/tdyoshida/new_paquo_test/QuPath-0.4.4.app
paquo config -l

And post the output here. My current guess is, that for some reason either a different python or a different QuPath is used. The output should help us debug further.

One thing I noticed is that I cannot open the newly downloaded QuPath from Mac GUI. It showed an error that The application "QuPath-0.4.4" can't be opened. Is it an expected behavior?

That's the same for me. I think this is due to a Mac security "feature". You can work around it by running chmod +x QuPath-0.4.4.app/Contents/MacOS/QuPath in the new_paquo_test folder. Then you can start QuPath normally. But this should not impact running paquo with it.

[tdyoshida]

Hi @ap-- ,

Here's the output:

$ cd ~/new_paquo_test
$ conda activate paquo_test
$ which python
/opt/homebrew/Caskroom/miniconda/base/envs/paquo_test/bin/python

$ export PAQUO_QUPATH_DIR=/Users/tdyoshida/new_paquo_test/QuPath-0.4.4.app
$ paquo config -l
# current paquo configuration
# ===========================
# format: TOML
qupath_dir = "/Users/tdyoshida/new_paquo_test/QuPath-0.4.4.app"
qupath_search_dirs = [
    "/opt",
    "/Applications",
    "c:/Program Files",
    "/usr/local",
    "~/Applications",
    "~/AppData/Local",
    "~",
]
qupath_search_dir_regex = "(?i)qupath.*"
qupath_search_conda = true
qupath_prefer_conda = true
java_opts = [
    "-XX:MaxRAMPercentage=50",
]
safe_truncate = true
jvm_path_override = ""
mock_backend = false
cli_force_log_level_error = true
warn_microsoft_store_python = true

Although I have no idea what is the problem, it must be due to something I did outside of my conda environment. paquo was working no problem until 2-3 weeks ago.

Thank you so much for taking your time for the problem that might not be an issue of paquo itself.

[ap--]

I don't really know why this is not working for you...

Do you have JAVA_HOME configured in your environment? If yes, can you try to unset it?

unset JAVA_HOME

In you log it says: JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-18.0.2.1.jdk/Contents/Home Did you install this manually?

[tdyoshida]

After unset JAVA_HOME, there's still the same error :(

I actually don't remember exactly, but I must have set $JAVA_HOME to build something that depends on JDK18... All I can tell for sure is that paquo was working even after the $JAVA_HOME was set with this variable.

For now my practical solution is to use paquo within Docker container. It is not a perfect solution for me, but I was able to build an image with paquo and QuPath (I took your code to download QuPath within Docker).

I leave the decision to you whether to close this issue (hopefully somebody in the future can find a solution). Thank you so much!

[tdyoshida]

Hi @ap-- ,

Actually I was able to figure out what caused the problem (I'm no expert to tell why this was the problem, though).

I use neovim, and there is a plugin called image.nvim, which adds image support. It requires ImageMagick, and the installation instruction asks to change $DYLD_LIBRARY_PATH.

Installing ImageMagick

The magick luarock provides bindings to ImageMagick's MagickWand, so we need to install that package as well.

Ubuntu: sudo apt install libmagickwand-dev MacOS: brew install imagemagick By default, brew installs into a weird location, so you have to add $(brew --prefix)/lib to DYLD_LIBRARY_PATH by adding something like export DYLD_LIBRARY_PATH="$(brew --prefix)/lib:$DYLD_LIBRARY_PATH" to your shell profile (probably .zshrc or .bashrc)

It looks like this caused the problem.

$ echo $DYLD_LIBRARY_PATH
# /opt/homebrew/lib:
$ conda activate paquo_test
$ python -c "import paquo.projects" && echo "OK"
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x0000000000000000, pid=49348, tid=259
#
# JRE version:  (17.0.8.1+1) (build )
# Java VM: OpenJDK 64-Bit Server VM (17.0.8.1+1, mixed mode, tiered, compressed class ptrs, g1 gc, bsd-aarch64)
# Problematic frame:
# C  0x0000000000000000
#
# Core dump will be written. Default location: /cores/core.49348
#
# An error report file with more information is saved as:
# /Users/tdyoshida/GitHub/dotfiles/hs_err_pid49348.log
#
#
[1]    49348 abort      python -c "import paquo.projects"
$ echo $DYLD_LIBRARY_PATH
# /opt/homebrew/lib:
$ unset DYLD_LIBRARY_PATH
$ python -c "import paquo.projects" && echo "OK"
# OK

I tried to install the above plugin, and later decided not to use it. However, I left the configuration in my .zshrc, without thinking it might cause a problem elsewhere...

When I ran the script you provided, it ran from bash, and the above config is in my .zshrc. Thus there was no error.

Thank you so much for spending your time!

[ap--]

I'm happy that you have found the solution! 😊

I don't think it makes sense for paquo as a library to detect environment changes like this and warn the user about potential errors. But this issue will help in case people search for a similar error.

If you could post your docker file that might be useful for others as a clean check when debugging their issues.

Have a great weekend! Cheers Andreas

[tdyoshida]

Here's an example Dockerfile.

FROM --platform=linux/amd64 continuumio/anaconda3:latest

RUN apt update && apt install -y \
  wget \
  make \
  gcc \
  git \
  unzip \
  build-essential \
  libgtk-3-dev

# Create .bash_profile and load bashrc
RUN echo "if [ -f ~/.bashrc ]; then  . ~/.bashrc;  fi" >> ~/.bash_profile

RUN conda install conda-forge::paquo
RUN paquo get_qupath --download-path . --install-path . 0.4.4 | tee .qupath_install_path
RUN echo "export PAQUO_QUPATH_DIR=$(grep -v "^#" < .qupath_install_path)" >> ~/.bashrc

RUN mkdir -p /Volumes/data

CMD ["/bin/bash"]

As image file paths are stored in the QuPath project file (e.g. "/Volumes/data/project_dir/image_file"), a directory is created in the container ("/Volumes/data"), then the local volume is mounted as docker run ... -v /Volumes/data:/Volumes/data .... This part should be changed based on each user's environment. Alternatively, the user can change the image file paths using QuPathProject.update_image_paths inside the container.