Thread Safety of PyCall with ibm db package

[chowerth]

I'm having an issue using threads with PyCall. I'm using IBM's db2 python package trying to execute queries on multiple threads and it will crash 99% of the time (race condition?). Serial code always works. I've tried using Threads, Dagger, and Distributed and all have the same issue. I read that disabling faulthandler might fix it but I think I'm properly disabling it and it still occurs for me? Any help is appreciated.

Example with @spawn crashing:

Minimalist example showing it crashing:

[the-noble-argon]

I'm having the exact same issue. I wrote a minimalist version of this code that just writes a simple JSON file that can reliably crash. I even tried applying a lock to anything that uses Python so that the code executes in a serial manner, but I still get these crashes.

The code:

using PyCall
const PY_LOCK = ReentrantLock()
const PY_JSON = pyimport("json")

function write_json_file(fileName::String, outputData::Dict)
    outputFile = open(fileName, "w")
    lock(PY_LOCK)
    try
        PY_JSON.dump(outputData, outputFile)
    finally
        unlock(PY_LOCK)
        close(outputFile)
    end

    return nothing
end

function file_operation(fileName::String)
    outputData = Dict("a"=>randn(), "b"=>randn())

    write_json_file(fileName, outputData)
    return outputData
end

file_operation("testfile.json")
firstTask = Threads.@spawn log.(exp.(randn(10000000)))
taskList = [Threads.@spawn file_operation("testfile$(ii).json") for ii in 1:30]

the error message:

Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x7ffa6e17001c -- PyDict_GetItem at C:\Users\user\Miniconda3\python39.dll (unknown line)
in expression starting at none:0
PyDict_GetItem at C:\Users\user\Miniconda3\python39.dll (unknown line)
PyObject_GetAttrString at C:\Users\user\Miniconda3\python39.dll (unknown line)
_getproperty at C:\Users\user\.julia\packages\PyCall\ygXW2\src\PyCall.jl:300
__getproperty at C:\Users\user\.julia\packages\PyCall\ygXW2\src\PyCall.jl:312 [inlined]
getproperty at C:\Users\user\.julia\packages\PyCall\ygXW2\src\PyCall.jl:318
write_json_file at g:\My Drive\julia_pycall_threadjson.jl:10
file_operation at g:\My Drive\julia_pycall_threadjson.jl:23
#4 at .\threadingconstructs.jl:258
unknown function (ip: 0000000061883543)
jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1838 [inlined]
start_task at /cygdrive/c/buildbot/worker/package_win64/build/src\task.c:931
Allocations: 10329372 (Pool: 10322778; Big: 6594); GC: 12