How to use it with multithreading?

[untoreh]

I am using nimpy to call into python across multiple threads. I hold a lock whenever I use python functions, such that only one thread at a time calls into python. But this doesn't seem to be enough. I get segfaults with a traceback pointing to the pyobject destructor. So I assume that holding a lock is not enough to be GIL safe, because the destructor implemented in nimpy is not GIL safe.

I tried to hold the lock myself with this, but in my code I got stalls, maybe because the PyGILState_Ensure call stops the nim threading runtime I don't know:

type PyGILState_STATE = enum PyGILState_LOCKED, PyGILState_UNLOCKED

{.pragma: pyfunc, cdecl, gcsafe.}
initPyLibIfNeeded()
let
    m = pythonLibHandleForThisProcess()
    PyGILState_Ensure = cast[proc(): PyGILState_STATE {.pyfunc.}](m.symAddr("PyGILState_Ensure"))
    PyGILState_Release = cast[proc(s: PyGILState_STATE) {.pyfunc.}](m.symAddr("PyGILState_Release"))

template withGIL*(code) =
    let state = Py_GILState_Ensure()
    code
    Py_GILState_Release(state)

type GilLock* = object
        m: LibHandle
        s: PyGILState_STATE

proc initGilLock*(): ptr GilLock =
    result = create(GilLock)
    result.m = pythonLibHandleForThisProcess()

proc acquire*(g: ptr GilLock) = g.s = Py_GILState_Ensure()
proc release*(g: ptr GilLock) = Py_GILState_Release(g.s)

In my case I ended up using threadvars, but I don't know why it works, even if the vars are globals, shouldn't a new assignment trigger a destructor on the previous object?

[yglukhov]

Have you tried --gc:orc?

[untoreh]

From what I gathered the problem comes from not checking that the call into python tp_dealloc is made while the current thread is NOT NULL. Seems to be working by changing the destructor to this:

  proc `=destroy`*(p: var PyObject) =
    if not p.rawPyObj.isNil:
      let ts = PyThreadState_Swap(pyMainThread)
      decRef p.rawPyObj
      discard PyThreadState_Swap(ts)
      p.rawPyObj = nil

I initialize pyMainThread early right before the lib is loaded, such that no threadpool has yet started to get the main thread state.

[cvanelteren]

Is there a solution to this that can be achieved without modifying the source?

[cvanelteren]

For posterity, I used @untoreh fork of nimpy and used the with gil succesfuly, my own attempts were no releasing the gil properly.