No way to install Julia signal handler when other dependency uses juliacall

[MilesCranmer]

Affects: JuliaCall

Describe the bug

Related to https://github.com/JuliaPy/PythonCall.jl/issues/219. There's currently a scenario with no possible workaround other than forcing a downstream user to set up environment variables to avoid a segfault.

Say that I import a package which uses serial Julia code. That package runs

from juliacall import Main as jl

to start Julia.

Then, I import a different package. That package calls multithreaded Julia code, and thus to avoid a segfault (https://juliapy.github.io/PythonCall.jl/stable/faq/#Is-PythonCall/JuliaCall-thread-safe?) instead performs the import instead as:

import os
os.environ["PYTHON_JULIACALL_HANDLE_SIGNALS"] = "yes"

from juliacall import Main as jl

However, at this point, the Julia runtime has already started. So now when I run a multithreaded portion of code, I will get a segfault, even though I implemented special workarounds in the multithreaded package.

How could I avoid this scenario? Indeed the user could always set up the environment variables themselves. But managing to figure this out from a random bus error will only be ~5% of users.

[MilesCranmer]

My temporary workaround is to put this code:

if "juliacall" in sys.modules:
    warnings.warn(
        "juliacall module already imported. Make sure that you have set `PYTHON_JULIACALL_HANDLE_SIGNALS=yes` to avoid segfaults."
    )

before the import statement.

[cjdoris]

Well firstly, packages shouldn't generally be setting global config such as that env var, though I understand why you're doing it.

Over in #219 I suggested adding a warning to set PYTHON_JULIACALL_HANDLE_SIGNALS=yes if Julia is started with multiple threads. Do you think this would be a sufficient warning?

Also, given Julia starts with 1 thread by default, presumably you're already telling your users to set PYTHON_JULIACALL_NUM_THREADS? In which case you can also tell them to set PYHON_JULIACALL_HANDLE_SIGNALS?

[hameerabbasi]

If it helps, I was able to get a stacktrace for this:

Stacktrace

``` Stacktrace: [1] wait @ ./task.jl:352 [inlined] [2] open_exclusive(path::String; mode::UInt16, poll_interval::Int64, wait::Bool, stale_age::Int64, refresh::Float64) @ FileWatching.Pidfile ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:268 [3] open_exclusive @ ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:232 [inlined] [4] mkpidlock(at::String, pid::Int32; stale_age::Int64, refresh::Float64, kwopts::@Kwargs{}) @ FileWatching.Pidfile ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:67 [5] mkpidlock @ ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:62 [inlined] [6] mkpidlock(f::Pkg.Types.var"#51#54"{String, String, Dates.DateTime, String}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64}) @ FileWatching.Pidfile ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:91 [7] mkpidlock @ ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:90 [inlined] [8] mkpidlock @ ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:88 [inlined] [9] write_env_usage(source_file::String, usage_filepath::String) @ Pkg.Types ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/Pkg/src/Types.jl:539 [10] Pkg.Types.EnvCache(env::Nothing) @ Pkg.Types ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/Pkg/src/Types.jl:377 [11] EnvCache @ ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/Pkg/src/Types.jl:356 [inlined] [12] add_snapshot_to_undo(env::Nothing) @ Pkg.API ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/Pkg/src/API.jl:2189 [13] add_snapshot_to_undo @ ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/Pkg/src/API.jl:2185 [inlined] [14] activate(path::String; shared::Bool, temp::Bool, io::Base.DevNull) @ Pkg.API ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/Pkg/src/API.jl:1973 [15] top-level scope @ none:7 nested task error: InterruptException: Stacktrace: [1] poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) @ Base ./task.jl:985 [2] wait() @ Base ./task.jl:994 [3] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) @ Base ./condition.jl:130 [4] wait @ ./condition.jl:125 [inlined] [5] _trywait(t::Timer) @ Base ./asyncevent.jl:142 [6] wait @ ./asyncevent.jl:159 [inlined] [7] sleep(sec::Int64) @ Base ./asyncevent.jl:265 [8] (::FileWatching.Pidfile.var"#13#14"{Int64, String})() @ FileWatching.Pidfile ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:263 caused by: IOError: FileMonitor (start): no such file or directory (ENOENT) Stacktrace: [1] uv_error @ ./libuv.jl:100 [inlined] [2] start_watching(t::FileWatching.FileMonitor) @ FileWatching ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/FileWatching.jl:525 [3] wait(m::FileWatching.FileMonitor) @ FileWatching ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/FileWatching.jl:659 [4] watch_file(s::String, timeout_s::Float64) @ FileWatching ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/FileWatching.jl:772 [5] watch_file @ ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/FileWatching.jl:778 [inlined] [6] (::FileWatching.Pidfile.var"#13#14"{Int64, String})() @ FileWatching.Pidfile ~/miniforge3/envs/sparse/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:260 ```

This can also be avoided by setting the environment variable PYTHONFAULTHANDLER to some writable file. Credit to @willow-ahrens for some of the heavy lifting on this, cc @mtsokol.