[BUG]: Runtime error loading `numpy`

nkr0 commented 4 months ago

Required prerequisites

[X] Make sure you've read the documentation. Your issue may be addressed there.
[X] Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
[X] Consider asking first in the Gitter chat room or in a Discussion.

What version (or hash if on master) of pybind11 are you using?

2.12.0

Problem description

All the python parts happen inside a function with a scoped_interpreter. I call the function twice from main. The first time, everything works. The second time it fails. And this is only if numpy is imported. Output error is different in different versions of python and numpy. But I think it is related to numpy's CPU dispatcher.

Reproducible example code

#include <iostream>
#include <pybind11/embed.h>

namespace py = pybind11;

void test() {
  py::scoped_interpreter guard;
  try {
    py::exec(R"(
      import numpy
      print("module loaded")
    )");
  } catch (py::error_already_set &e) {
    std::cerr << e.what() << "\n";
  }
}

int main() {
  // py::scoped_interpreter guard; // If the `guard` is in the `main` scope, there is no issue.
  test();
  test(); // this call fails to import numpy
  return 0;
}

Is this a regression? Put the last known working version here if it is.

Not a regression

AYESDIE commented 4 months ago

I face a similar issue. It occurs not only with the use of numpy but also any libraries that uses numpy. For example cv2.

SystemError: Objects/structseq.c:481: bad argument to internal function

 At:
   /usr/lib/python3.10/site-packages/numpy/core/overrides.py(8): <module>
   <frozen importlib._bootstrap>(241): _call_with_frames_removed
   <frozen importlib._bootstrap_external>(883): exec_module
   <frozen importlib._bootstrap>(703): _load_unlocked
   <frozen importlib._bootstrap>(1006): _find_and_load_unlocked
   <frozen importlib._bootstrap>(1027): _find_and_load
   <frozen importlib._bootstrap>(241): _call_with_frames_removed
   <frozen importlib._bootstrap>(1079): _handle_fromlist
   /usr/lib/python3.10/site-packages/numpy/core/multiarray.py(10): <module>
   <frozen importlib._bootstrap>(241): _call_with_frames_removed
   <frozen importlib._bootstrap_external>(883): exec_module
   <frozen importlib._bootstrap>(703): _load_unlocked
   <frozen importlib._bootstrap>(1006): _find_and_load_unlocked
   <frozen importlib._bootstrap>(1027): _find_and_load
   <frozen importlib._bootstrap>(241): _call_with_frames_removed
   <frozen importlib._bootstrap>(1079): _handle_fromlist
   /usr/lib/python3.10/site-packages/numpy/core/__init__.py(52): <module>
   <frozen importlib._bootstrap>(241): _call_with_frames_removed
   <frozen importlib._bootstrap_external>(883): exec_module
   <frozen importlib._bootstrap>(703): _load_unlocked
   <frozen importlib._bootstrap>(1006): _find_and_load_unlocked
   <frozen importlib._bootstrap>(1027): _find_and_load
   <frozen importlib._bootstrap>(241): _call_with_frames_removed
   <frozen importlib._bootstrap>(992): _find_and_load_unlocked
   <frozen importlib._bootstrap>(1027): _find_and_load
   /usr/lib/python3.10/site-packages/numpy/__config__.py(4): <module>
   <frozen importlib._bootstrap>(241): _call_with_frames_removed
   <frozen importlib._bootstrap_external>(883): exec_module
   <frozen importlib._bootstrap>(703): _load_unlocked
   <frozen importlib._bootstrap>(1006): _find_and_load_unlocked
   <frozen importlib._bootstrap>(1027): _find_and_load
   /usr/lib/python3.10/site-packages/numpy/__init__.py(131): <module>
   <frozen importlib._bootstrap>(241): _call_with_frames_removed
   <frozen importlib._bootstrap_external>(883): exec_module
   <frozen importlib._bootstrap>(703): _load_unlocked
   <frozen importlib._bootstrap>(1006): _find_and_load_unlocked
   <frozen importlib._bootstrap>(1027): _find_and_load
   /usr/lib/python3.10/site-packages/cv2/__init__.py(13): <module>
   <frozen importlib._bootstrap>(241): _call_with_frames_removed
   <frozen importlib._bootstrap_external>(883): exec_module
   <frozen importlib._bootstrap>(703): _load_unlocked
   <frozen importlib._bootstrap>(1006): _find_and_load_unlocked
   <frozen importlib._bootstrap>(1027): _find_and_load
   /development/numpy-api/examples/cv2.py(1): <module>
   <frozen importlib._bootstrap>(241): _call_with_frames_removed
   <frozen importlib._bootstrap_external>(883): exec_module
   <frozen importlib._bootstrap>(703): _load_unlocked
   <frozen importlib._bootstrap>(1006): _find_and_load_unlocked
   <frozen importlib._bootstrap>(1027): _find_and_load

cv2.py:

import cv2

def dummy():
    print("CV2 imported")

My setup is pretty much similar to what @nkr0 has reported here with the only difference being I'm running my python interpreter in a qt application. So when I try to run this part of the code more than once, I encounter this issue.

nkr0 commented 4 months ago

@AYESDIE I think it is exactly the same issue. I too found it first in a Qt app and then created a minimal example for creating an issue. You've a slightly different error because you are using python 3.10 and, I'm guessing, numpy 1.x. The error I've is from numpy 2.x which was released only this week. With numpy 1.x, I've the same error. Also, for indirect imports through cv2, matplotlib, etc.

nkr0 commented 4 months ago

I tested not using pybind11 and got the same results

#include <Python.h>

void test() {
  Py_Initialize();
  // const auto gstate = PyGILState_Ensure(); // this has no effect on the error
  PyRun_SimpleString(R"(
import numpy
print("module loaded")
    )");
  // PyGILState_Release(gstate);
  Py_Finalize();
}

int main() {
  // Works if `Py_Initialize` and `Py_Finalize` are in the `main` scope.
  // Py_Initialize();
  test();
  test(); // RuntimeError: CPU dispatcher tracer already initlized
  // Py_Finalize();
  return 0;
}

Searching further, I found these posts https://stackoverflow.com/questions/59314230/python-c-api-crashes-on-import-numpy-when-initilizing-multiple-times https://stackoverflow.com/questions/14843408/python-c-embedded-segmentation-fault https://stackoverflow.com/questions/7676314/py-initialize-py-finalize-not-working-twice-with-numpy https://stackoverflow.com/questions/74862962/exception-thrown-when-using-python-in-for-loop-in-c-script https://stackoverflow.com/questions/7676314/py-initialize-py-finalize-not-working-twice-with-numpy and this caveat in https://docs.python.org/3/c-api/init.html#c.Py_FinalizeEx

Some extensions may not work properly if their initialization routine is called more than once; this can happen if an application calls Py_Initialize() and Py_FinalizeEx() more than once.

AYESDIE commented 4 months ago

From what I recall while using Numpy C API and CPython API, there is this function import_array() which needs to be called once before using Numpy in the C++ code. I can also recall that calling this import_array() multiple times would result in some issues.

nkr0 commented 4 months ago

That sounds like if you are using Numpy's C API. But here we are using CPython API. Maybe I'm missing something. Anyway, I wanted to create multiple scoped_interpreters to get clean python sessions. And now from reading it seems like calling Py_Initialize more than once in a process is not a good idea. But, the following workaround is enough for my case.

#include <pybind11/embed.h>

namespace py = pybind11;

void test(const char *code) {
  // Clear references to modules and variables from `__main__`
  PyDict_Clear(PyModule_GetDict(PyImport_AddModule("__main__")));
  py::exec(R"(
import numpy
print("module loaded")
    )");
  py::exec(code);
}

int main() {
  py::scoped_interpreter guard; 
  test("x=1; print('First time', x)");
  // I want this to fail with `NameError: name 'x' is not defined`
  test("print('Second time', x)");
  return 0;
}

scoped_interpreter is in the main scope; which solves the issue with import numpy. And PyDict_Clear helps clean variables from the interpreter. Not at all thread safe.

Issue can be closed if pybind does not want to make any changes to address this.

pybind / pybind11