Updating pybind and ImportError: dynamic module

[davidebenato]

my code uses a very old version of pybind: 2.6.0 and everything works fine. I have two modules, liba and libb

With 2.6.0 I have:

>>> import liba
>>> import libb
>>> 

Same code on 2.7.1 and higher, and I get the following error. I didn't change one single line in the code. Importing liba still works, but libb fails.

>>> import liba
>>> import libb
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dynamic module does not define module export function (PyInit_libb)

My module is correctly defined in a cpp file.

PYBIND11_MODULE(libb, m)

My CMake looks like this:

if (BUILD_SHARED_LIBRARY)
    set(TARGET_LINK_LIBS liba-shared Eigen3::Eigen)
    target_link_libraries(objlib PUBLIC ${TARGET_LINK_LIBS})
    find_package(Python REQUIRED COMPONENTS Interpreter Development)
    include_directories("${Python_INCLUDE_DIRS}")

    add_library(objlib-py OBJECT src/python/libb.cpp)
    SET(TARGET_LINK_LIBS_PY Python::Module pybind11::pybind11 ${TARGET_LINK_LIBS})
    target_include_directories(objlib-py PUBLIC ${TARGET_INCLUDE_DIRS})
    target_link_libraries(objlib-py PUBLIC ${TARGET_LINK_LIBS_PY})
    set_property(TARGET objlib-py PROPERTY POSITION_INDEPENDENT_CODE ON)

    set(SHARED_LIB ${CMAKE_PROJECT_NAME})
    add_library(${SHARED_LIB} SHARED $<TARGET_OBJECTS:objlib> $<TARGET_OBJECTS:objlib-py>)
    target_compile_definitions(${SHARED_LIB} PRIVATE VERSION_INFO=${PROJECT_VERSION})
    target_include_directories(${SHARED_LIB} PUBLIC ${TARGET_INCLUDE_DIRS})
    target_link_libraries(${SHARED_LIB} PUBLIC ${TARGET_LINK_LIBS_PY})
    list(APPEND RISK_TARGETS ${SHARED_LIB})
endif (BUILD_SHARED_LIBRARY)

And my environment:

$ python3 --version
Python 3.9.2

$ python3
Python 3.9.2 (default, Feb 28 2021, 17:03:44) 
[GCC 10.2.1 20210110] on linux

I compile my libs with gcc12

I am puzzled why upgrading the version of pybind makes my imports fail. Maybe I am doing something wrong, but I can't pinpoint it.

[henryiii]

liba needs PYBIND11_MODULE(liba, m), and libb needs PYBIND11_MODULE(libb, m), not sure where librisk comes in. Do you have a source link?

[davidebenato]

I indeed have:

PYBIND11_MODULE(liba, m),

in liba and

PYBIND11_MODULE(libb, m)

in libb

I edited the original question, it was just to anonymize the code. Unfortunately I can't share the full code as it is private. I am confident that my code is fine on 2.6.0, we had it in production for years. Has anything changed since then and do I need to update anything?

Sorry for the lack of details, I tried to gather as much information as I could.

[henryiii]

Is there an issue finding Python? If it finds Python 2, for example, the above would happen. Ahh, one big change - you are manually calling FindPython, so pybind11 will use that nowadays, rather than using the deprecated (and removed, sort of, in CMake 3.27) FindPythonInterp. Are you calling find_package(Python) before find_package(pybind11)? FYI, that's not quite the right way to call FindPython, you should use Development.Module instead of Development (at least if you want to make wheels one day).

(Also, you should use targets everywhere instead if classic statements like include_directories("${Python_INCLUDE_DIRS}"), but unrelated. Also pybind11::pybind11 already includes Python::Module, IIRC).

[davidebenato]

Thank for the help. I am using python3, and I have no issues in finding python for compilation. It happily compiles the library.

in liba I have

    find_package(Python 3.8 REQUIRED COMPONENTS Interpreter Development)
    include_directories("${Python_INCLUDE_DIRS}")
    find_package(pybind11 2.6.0 REQUIRED)

in libb I don't find_package pybind11 because it's pulled from liba Like follows

    find_package(liba REQUIRED)
    find_package(Python REQUIRED COMPONENTS Interpreter Development)

So I guess they are reversed in this case. Could that cause issues?

I'll try adding Development.Module instead of Development

[davidebenato]

I tried with reversing the order in libb:

find_package(Python REQUIRED COMPONENTS Interpreter Development.Module)
find_package(liba REQUIRED)

but the problem persists:

When I import libb

$ python3
Python 3.9.2 (default, Feb 28 2021, 17:03:44) 
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import liba
>>> import libb
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dynamic module does not define module export function (PyInit_libb)