Missing working example for ctypes dll building

[mkielg11]

Hello

I am trying to make a minimum working example of automatic building of a standard c-code library for import with ctypes, as per this post stack-overflow post: https://stackoverflow.com/questions/62388018/installing-pure-c-library-for-python-package Someone suggested this package, and I have been trying to make it work, as per this branch: https://github.com/mkielg11/ctypes_distribution/tree/fix-using-setuptoolsdso

However, I have two issues:

1. The build .so/.dll does not seem to be working with ctypes when loading the library
2. The build .so/.dll is not included when installing the package using pip

I though maybe you could help making a viable example of how to use your package?

[mdavidsaver]

Have you seen my example? https://github.com/mdavidsaver/setuptools_dso/tree/master/example

Your stackoverflow post looks like Windows, but I don't see any dllimport/export in your code. cf. the usage of MYLIB_API

https://github.com/mdavidsaver/setuptools_dso/blob/master/example/src/mylib.h

[mkielg11]

Thanks for your quick response!

I have seen your example, but in my use case, I am trying to include c-code which is not written for the Python package, i.e. I cannot add includes in the header file. As of now, I can build the .so file with a make-file and include it with c-types, however, that makes people dependent om makefile and the related dependencies.

I would like it to work on both linux and windows, as it's a shared package.

[mdavidsaver]

I am trying to include c-code which is not written for the Python package

dllimport/export are not specific to Python or setuptools_dso. This is the MS recommended way to define the external interface of a DLL.

I cannot add includes in the header file.

Fair enough, this is a common constrain when packaging.

I can build the .so file with a make-file and include it with c-types

I think the place for you to start is understanding what the Makefile is doing. It is likely passing some extra flags to the compiler and/or linker. Or perhaps a .def file is used for the DLL build?

Once these additional inputs are defined, they can be passed to DSO(..., extra_compile_args=[...], extra_link_args=[...])

eg. for GCC/mingw you might look for -Wl,--export-all-symbols being passed to the linker.

[mkielg11]

Thanks for the helpful reply!

I've worked a bit to find out what I'm missing. I've made a reference Makefile that can produce the desired shared library. Doing so, I believe I've found out that the required linker argument is the "-shared" keyword, and with the modifications I've made to the setup.py, adding compiler and linker arguments for the DSO setup, I have made it work on linux, which is great!

However, while I with the Makefile using minGW can make libraries for Windows, but when using Microsoft Build Tools (called via setup.py / setuptools_dso), I have found out that the shared argument is not support by this compiler, so the dll generated when calling from Windows does not work. During build, I get: LINK : warning LNK4044: unrecognized option '/shared'; ignored Any ideas on how to get around this? I've tried searching for an equivalent option for the Build Tools compiler with no success yet.

[mdavidsaver]

Oh dear, Cygwin... I should have asked explicitly about your environment. I don't test with cygwin, so I'm not surprised that you are encountering problems. I should warn you up front that adding support for cygwin would likely to be an involved process, and might not even be possible. Some of the tricks I play may conflict with what Cygwin does to partially emulate ELF loader behavior on Windows.

Would you be able to switch to native mingw or MSVC, or WSL, or really anything else? I think you will have a happier life in general without cygwin (I have ;) ).

fyi. -shared is the GCC/binutils flag has asks the linker to create a shared library, otherwise the default is to link an executable. MSVC does this differently. You shouldn't need to explicitly pass -shared as setuptools_dso (really distutils) should be adding this for you. Unsurprisingly though, cygwin is a special case.

If you want to pursue cygwin further, please attach the full output of eg. ./setup.py -v bdist_wheel for your setup.py, and for comparison some other python module with a C extension which you are able to successfully build and run.

[mkielg11]

You're right! I must just not have tested it cleanly in linux.

Okay, so my current issue is, that it works without any compile-flags at all when installing the package using setup.py in linux, however, I get the following error when installing in windows (compiled with setuptools_dso, i.e. MSVC):

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Anaconda3\envs\py3.8\lib\site-packages\xpackage\xpackage.py", line 22, in xpackage_function
    get_magic_number_func = xlib.return_magic_number
  File "C:\Anaconda3\envs\py3.8\lib\ctypes\__init__.py", line 394, in __getattr__
    func = self.__getitem__(name)
  File "C:\Anaconda3\envs\py3.8\lib\ctypes\__init__.py", line 399, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: function 'return_magic_number' not found

So something is not working with the MSVC compiler. I do try to minimise my use of cygwin, which is some of the reason making this work would be great - I then won't have to add another dependency to the team needing the package(s). As far as I see, cygwin should not be involved in any of the above? With the verbose output, I get:

python ./setup.py -v bdist_wheel
running bdist_wheel
running build
running build_py
creating build
creating build\lib
creating build\lib\xpackage
copying xpackage\xpackage.py -> build\lib\xpackage
copying xpackage\__init__.py -> build\lib\xpackage
creating build\lib\xpackage\clib
copying xpackage\clib\cfile_dsoinfo.py -> build\lib\xpackage\clib
copying xpackage\clib\__init__.py -> build\lib\xpackage\clib
running build_dso
Building DSOs
building 'xpackage.clib.cfile' DSO as build\lib\xpackage\clib\cfile.dll
creating build\temp.win-amd64-3.8
creating build\temp.win-amd64-3.8\Release
creating build\temp.win-amd64-3.8\Release\xpackage
creating build\temp.win-amd64-3.8\Release\xpackage\deps
creating build\temp.win-amd64-3.8\Release\xpackage\deps\xclib
C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.27.29110\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Ixpackage/deps/xclib/include/ "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.27.29110\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt" /Tcxpackage/deps/xclib/cfile.c /Fobuild\temp.win-amd64-3.8\Release\xpackage/deps/xclib/cfile.obj
cfile.c
Executing "C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.27.29110\bin\HostX86\x64\link.exe" /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.27.29110\lib\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.19041.0\ucrt\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.19041.0\um\x64 build\temp.win-amd64-3.8\Release\xpackage/deps/xclib/cfile.obj /OUT:build\lib\xpackage\clib\cfile.dll /IMPLIB:build\lib\xpackage\clib\cfile.lib
C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.27.29110\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.27.29110\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.19041.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.19041.0\um\x64" build\temp.win-amd64-3.8\Release\xpackage/deps/xclib/cfile.obj /OUT:build\lib\xpackage\clib\cfile.dll /IMPLIB:build\lib\xpackage\clib\cfile.lib
Generating code
Finished generating code
creating info module for xpackage.clib.cfile at build\lib\xpackage\clib\cfile_dsoinfo.py
installing to build\bdist.win-amd64\wheel
running install
running install_lib
creating build\bdist.win-amd64
creating build\bdist.win-amd64\wheel
creating build\bdist.win-amd64\wheel\xpackage-0.0.5.data
creating build\bdist.win-amd64\wheel\xpackage-0.0.5.data\purelib
creating build\bdist.win-amd64\wheel\xpackage-0.0.5.data\purelib\xpackage
creating build\bdist.win-amd64\wheel\xpackage-0.0.5.data\purelib\xpackage\clib
copying build\lib\xpackage\clib\cfile.dll -> build\bdist.win-amd64\wheel\xpackage-0.0.5.data\purelib\xpackage\clib
copying build\lib\xpackage\clib\cfile_dsoinfo.py -> build\bdist.win-amd64\wheel\xpackage-0.0.5.data\purelib\xpackage\clib
copying build\lib\xpackage\clib\__init__.py -> build\bdist.win-amd64\wheel\xpackage-0.0.5.data\purelib\xpackage\clib
copying build\lib\xpackage\xpackage.py -> build\bdist.win-amd64\wheel\xpackage-0.0.5.data\purelib\xpackage
copying build\lib\xpackage\__init__.py -> build\bdist.win-amd64\wheel\xpackage-0.0.5.data\purelib\xpackage
running install_egg_info
running egg_info
writing xpackage.egg-info\PKG-INFO
writing dependency_links to xpackage.egg-info\dependency_links.txt
writing requirements to xpackage.egg-info\requires.txt
writing top-level names to xpackage.egg-info\top_level.txt
'license_file' option was not specified
reading manifest file 'xpackage.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*.cpp' under directory 'xpackage'
writing manifest file 'xpackage.egg-info\SOURCES.txt'
Copying xpackage.egg-info to build\bdist.win-amd64\wheel\xpackage-0.0.5.data\purelib\xpackage-0.0.5-py3.8.egg-info
Copying dependency_links.txt to build\bdist.win-amd64\wheel\xpackage-0.0.5.data\purelib\xpackage-0.0.5-py3.8.egg-info\dependency_links.txt
Copying not-zip-safe to build\bdist.win-amd64\wheel\xpackage-0.0.5.data\purelib\xpackage-0.0.5-py3.8.egg-info\not-zip-safe
Copying PKG-INFO to build\bdist.win-amd64\wheel\xpackage-0.0.5.data\purelib\xpackage-0.0.5-py3.8.egg-info\PKG-INFO
Copying requires.txt to build\bdist.win-amd64\wheel\xpackage-0.0.5.data\purelib\xpackage-0.0.5-py3.8.egg-info\requires.txt
Copying SOURCES.txt to build\bdist.win-amd64\wheel\xpackage-0.0.5.data\purelib\xpackage-0.0.5-py3.8.egg-info\SOURCES.txt
Copying top_level.txt to build\bdist.win-amd64\wheel\xpackage-0.0.5.data\purelib\xpackage-0.0.5-py3.8.egg-info\top_level.txt
running install_scripts
C:\Anaconda3\envs\py3.8\lib\site-packages\wheel\bdist_wheel.py:80: RuntimeWarning: Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect
  if get_flag('Py_DEBUG',
Original Wheel Tag: cp38, cp38, win_amd64
adding license file "LICENSE" (matched pattern "LICEN[CS]E*")
Original Wheel Tag: cp38, cp38, win_amd64
creating build\bdist.win-amd64\wheel\xpackage-0.0.5.dist-info\WHEEL
creating 'dist\xpackage-0.0.5-cp38-cp38-win_amd64.whl' and adding 'build\bdist.win-amd64\wheel' to it
adding 'xpackage-0.0.5.data/purelib/xpackage/__init__.py'
adding 'xpackage-0.0.5.data/purelib/xpackage/xpackage.py'
adding 'xpackage-0.0.5.data/purelib/xpackage/clib/__init__.py'
adding 'xpackage-0.0.5.data/purelib/xpackage/clib/cfile.dll'
adding 'xpackage-0.0.5.data/purelib/xpackage/clib/cfile_dsoinfo.py'
adding 'xpackage-0.0.5.dist-info/LICENSE'
adding 'xpackage-0.0.5.dist-info/METADATA'
adding 'xpackage-0.0.5.dist-info/WHEEL'
adding 'xpackage-0.0.5.dist-info/top_level.txt'
adding 'xpackage-0.0.5.dist-info/RECORD'
removing build\bdist.win-amd64\wheel

Would it help if I add the DSO to an Extension instead? What would be required then?

[mkielg11]

@mdavidsaver, any idea as to what is missing for getting the MSVC compiler to work?

[mkielg11]

I think I might be running into the issue discussed here (or at least related to it): https://stackoverflow.com/questions/225432/export-all-symbols-when-creating-a-dll/32284832#32284832

Currently, I have difficulties seeing a way to have an automated way of building this using the setuptools_dso (and MSVC in general), unless setuptools_dso could help automatically generating a .def file for the linker to export all symbols or something like that.

[mdavidsaver]

I agree that what you, and that stackoverflow chain, describe is what happens when a symbol is not exported (by dllexport decoration, .def file, or /EXPORT cli arg). cf. https://github.com/mdavidsaver/setuptools_dso/issues/20#issuecomment-931482589 above.