Segmentation fault when `juliacall` is imported before `torch`

[fiktor]

Minimal example:

$ python -c 'import juliacall; import torch; print("Done.")'
Segmentation fault (core dumped)

The segfault happens at 0x00007fffa673c342 in __kmp_env_blk_init () in libgomp.so.1

Importing only one of the two modules does not result in an error.

Note that this issue is related but different from #215: here juliacall is imported before torch, and there torch is imported before juliacall.

Environment details:

• Fedora release 37 (Thirty Seven)
• Python 3.10.11 (installed using a mamba environment)
• Torch version 1.11.0.post202 (pytorch-gpu, build cuda112py310h68407e5_202)
• JuliaCall version 0.9.15 (build pypi_0)
• Julia version 1.9.4

Similar issues:

• JuliaPy.#215: Heap corruption on Python when torch is imported before juliacall, but not the reverse FAQ
• JuliaPy.#518: Segmentation Fault When Using PyJulia Inside of PyTorch Custom Autograd Function

[fiktor]

Attached are other information which might be relevant to reproduce the bug: gdb trace and the list of installed packages (in plaintext and as environment.yml renamed environment.yml.txt to pass github filetype filter).

environment.yml.txt mamba_list.txt trace.txt

[fiktor]

It looks like this problem was mentioned in pytorch.#78829: while the issue 78829 is about the order "import torch; import juliacall", the comment cited above says

In the Linux scenario the issue occurs when Julia is loaded before pytorch within the same process.

which is consistent with this issue. Three more observations:

• The crash happens when the following line in torch/__init__.py is executed:

  198:    from torch._C import *  # noqa: F403

• Both juliacall and torch are loading the same libgomp.so.1
• There are some function names which are loaded by both juliacall and torch but exported by different libraries. Example: juliacall loads (indirectly) libopenblas64_.so and torch loads libmkl_gnu_thread.so.2. Both of these two libraries export symbol omp_in_parallel. I am not sure whether it is related to the segfault.

[mkitti]

You appear to be using juliacall which is unrelated to the pyjulia package. I'm going to transfer this issue to https://github.com/JuliaPy/PythonCall.jl

There are a number of possibilities here. You could for example configure Julia to use MKL.