apolukhin / Boost.DLL

Library for comfortable work with DLL and DSO
https://boost.org/libs/dll
110 stars 69 forks source link

Compatability with pybind11? #51

Closed danzimmerman closed 4 years ago

danzimmerman commented 4 years ago

Apologies in advance if this is actually expected behavior and the answer is "just don't try this."

I'm using the BOOST_DLL_ALIAS / boost_import_alias() machinery following this recipe to load a DLL library I developed in C++. It's been working great for me.

The library functionality is all in a single class Foo. There's a factory method Foo::Create() that returns a boost::shared_ptr to a new Foo, and a single BOOST_DLL_ALIAS in the project that exports Foo::Create() to a DLL symbol CreateFoo. The project is compiled to FooLibrary.dll and can be loaded and used from C++ exactly as described in the docs.

I want to be able to access the functionality from Python as well, so I defined a pybind11 interface using the PYBIND11_MODULE macro. The module's __init__ is bound to Foo::Create() though I've also tried it bound to a different factory function with the same results. When I compile with the pybind interface, I can copy and rename FooLibrary.dll to FooLibrary.pyd and import it as a correctly functioning module in Python.

However, compiling in the pybind code breaks it so that FooLibrary.dll can no longer be loaded as a C++ DLL. It fails with boost::dll::shared_library::load() failed: The specified module could not be found.

I'm using boost::dll::library_info in C++ to get the sections and symbols contained in the DLL and CreateFoo is in the DLL in both cases:

With only BOOST_DLL_ALIAS export in the source of FooLibrary.dll:

In C++, everything works perfectly, as expected:

--- Reading Symbols from shared library ---

Found no symbols in section .text
Found no symbols in section .rdata
Found no symbols in section .data
Found no symbols in section .pdata
Found symbol CreateFoo in section boostdll
Found no symbols in section .rsrc
Found no symbols in section .reloc

--- Trying boost::dll::import_alias() to load CreateDHSolver from builds/VS2019/x64/Release/FooLibrary ---

--- Shared Library Loaded Successfully, Program Continuing ---

In Python (copied compiled DLL and renamed to FooLibrary.pyd), there's the obvious failure:

Python 3.7.6 (default, Jan  8 2020, 20:23:39) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import FooLibrary
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dynamic module does not define module export function (PyInit_FooLibrary)

With both BOOST_DLL_ALIAS and PYBIND11_MODULE in the source of FooLibrary.dll:

In C++, it fails even though CreateFoo is an available symbol:

--- Reading Symbols from shared library ---

Found symbol PyInit_FooLibrary in section .text
Found no symbols in section .rdata
Found no symbols in section .data
Found no symbols in section .pdata
Found symbol CreateFoo in section boostdll
Found no symbols in section .rsrc
Found no symbols in section .reloc

--- Trying boost::dll::import_alias() to load CreateFoo from builds/VS2019/x64/Release/FooLibrary ---

boost::dll::shared_library::load() failed: The specified module could not be found

In Python, it works fine:

Python 3.7.6 (default, Jan  8 2020, 20:23:39) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import FooLibrary
>>> FooLibrary.Foo().doSomething()
'Foo Python Interface Did Something...'
>>>

Window 10, Microsoft Visual Studio 2019, Boost 1.7.2, C++17 language

Any ideas on how to resolve this would be appreciated. Or a clear understanding that it's impossible.

I expect this involves some fairly subtle point of C++ or DLL knowledge, and maybe I'll just have to live with compiling separate projects for the Python module and the C++ DLL.

The library is for use from C++ but needs some complex functional testing against Python prototype code, however, so it's really convenient to imagine having a dual-use Python/C++ DLL that has both interfaces in a single binary build. Is there a reason why boost::python would work better here?

danzimmerman commented 4 years ago

Sorry, will ask on main boost::dll repo