JVM crashes when trying to use a certain package in two separate threads with a SubInterpreter

[carlosuc3m]

Describe the bug I am trying to create different Subinterpreters in different threads. In each of those threads I need to import a Python package called "BioImageio.core" (here and here) . After importing the package in one thread, if I try to import it in another one I get the following warning:

c:\users\builder-admin\appdata\local\temp\abs_da_fnqccyg\croots\recipe\python-split_1661469486984\work\modules\_decimal\libmpdec\context.c:57: warning: mpd_setminalloc: ignoring request to set MPD_MINALLOC a second time

If I try then to call the libray, the JVM directly crashes:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00007ffed14ac759, pid=82736, tid=0x0000000000013b8c
#
# JRE version: Java(TM) SE Runtime Environment (8.0_271-b09) (build 1.8.0_271-b09)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.271-b09 mixed mode windows-amd64 compressed oops)
# Problematic frame:
# C  [_multiarray_umath.cp39-win_amd64.pyd+0x12c759]
#
# Failed to write core dump. Minidumps are not enabled by default on client versions of Windows
#
# An error report file with more information is saved as:
# C:\Users\hs_err_pid82736.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

Is there a way to avoid this? Is it related to the Python GIL?

REgards, CArlos

[bsteffensmeier]

Many third party modules are incompatible with sub-interpreters, the only workaround we can make within jep is to share the modules between interpreters using the shared modules field in JepConfig. In most cases many modules will need to be added to that list and it can be quite complicated tracking those down so we recommend using SharedInterpreters whenever possible. Your crash looks to be occurring in numpy which is known to be incompatible with subinterpreters.

[carlosuc3m]

Okk great. Thanks for the info!