search

pybind / pybind11

Seamless operability between C++11 and Python
https://pybind11.readthedocs.io/
Other
15.08k stars 2.05k forks source link

Runtime error loading `numpy` #5172

Closed nkr0 closed 1 week ago

nkr0 commented 1 week ago

I've this c++ code. It has import numpy in a function which is called twice.

#include <iostream>
#include <pybind11/embed.h>

namespace py = pybind11;

void test() {
  py::scoped_interpreter guard;
  try {
    py::exec(R"(
      import numpy
      print("module loaded")
    )");
  } catch (py::error_already_set &e) {
    std::cerr << e.what() << "\n";
  }
}

int main() {
  // py::scoped_interpreter guard; // If the `guard` is in the `main` scope, there is no issue.
  test();
  test(); // this call fails to import numpy
  return 0;
}

All the python parts happen inside a function with a scoped_interpreter. I call the function twice from main. The first time, everything works. The second time it fails. And this is only if numpy is imported. Output error is different in different version of python and numpy. But I think it is related to numpy's CPU dispatcher.

module loaded
RuntimeError: CPU dispatcher tracer already initlized

At:
  /home/nkr/.local/lib/python3.11/site-packages/numpy/_core/overrides.py(8): <module>
  <frozen importlib._bootstrap>(241): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(940): exec_module
  <frozen importlib._bootstrap>(705): _load_unlocked
  <frozen importlib._bootstrap>(1150): _find_and_load_unlocked
  <frozen importlib._bootstrap>(1176): _find_and_load
  <frozen importlib._bootstrap>(241): _call_with_frames_removed
  <frozen importlib._bootstrap>(1233): _handle_fromlist
  /home/nkr/.local/lib/python3.11/site-packages/numpy/_core/multiarray.py(10): <module>
  <frozen importlib._bootstrap>(241): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(940): exec_module
  <frozen importlib._bootstrap>(705): _load_unlocked
  <frozen importlib._bootstrap>(1150): _find_and_load_unlocked
  <frozen importlib._bootstrap>(1176): _find_and_load
  <frozen importlib._bootstrap>(241): _call_with_frames_removed
  <frozen importlib._bootstrap>(1233): _handle_fromlist
  /home/nkr/.local/lib/python3.11/site-packages/numpy/_core/__init__.py(51): <module>
  <frozen importlib._bootstrap>(241): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(940): exec_module
  <frozen importlib._bootstrap>(705): _load_unlocked
  <frozen importlib._bootstrap>(1150): _find_and_load_unlocked
  <frozen importlib._bootstrap>(1176): _find_and_load
  <frozen importlib._bootstrap>(241): _call_with_frames_removed
  <frozen importlib._bootstrap>(1126): _find_and_load_unlocked
  <frozen importlib._bootstrap>(1176): _find_and_load
  /home/nkr/.local/lib/python3.11/site-packages/numpy/__config__.py(4): <module>
  <frozen importlib._bootstrap>(241): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(940): exec_module
  <frozen importlib._bootstrap>(705): _load_unlocked
  <frozen importlib._bootstrap>(1150): _find_and_load_unlocked
  <frozen importlib._bootstrap>(1176): _find_and_load
  /home/nkr/.local/lib/python3.11/site-packages/numpy/__init__.py(115): <module>
  <frozen importlib._bootstrap>(241): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(940): exec_module
  <frozen importlib._bootstrap>(705): _load_unlocked
  <frozen importlib._bootstrap>(1149): _find_and_load_unlocked
  <frozen importlib._bootstrap>(1176): _find_and_load
  <string>(3): <module>

P.S. I had created a discussion when I thought that I was doing something wrong. But, now I'm more inclined to believe that this is an issue. Originally posted by @nkr0 in https://github.com/pybind/pybind11/discussions/5171