Guidelines for packaging projects using pybind11

[KerstinKeller]

Hi, I am greatly enjoying using pybind11. I am not sure if this question goes beyond the scope of this project, but I am sure I am not the only person dealing with this issue.

I have a mixed C++ / Python project, mainly a C++ library, a wrapper for this library using pybind11, some additional library functionality in pure python, using the pybind11 module, and lots of tests and usage samples in both C++ and Python.

So now, how can I properly package my library / SDK? The C++ part is easy, use CMake to build, and install/package. But how about my pure python files? Do I call Python from CMake or rather CMake from Python, or is it possible to stay in CMake entirely? The second link (it's the example linked in the documentation) unfortunately only builds the extension, but unfortunately does not distribute any other Python files with it.

How about my users? How would they like to install my module? On Ubuntu, standard way would be to install a .deb package for my C++ SDK, and then use pip or setup.py for the python part. On Windows? SDK installer and then build the extensions for a specific Python version? Deliver prebuild extensions and register them with existing Python installations?

As I said, this might be (and probably is) beyond the scope of this project, but some additional pointers in the documentation would be very helpful.

[bstaletic]

I don't knkow much about python packaging, but this comes down to teaching "setuptools" about the cmake part of the project.

[molpopgen]

With respect to the cmake_example repo, the gist of it would be something like:

1. mv src/ to cmake_example/src.
2. In setup.py, add packages=['cmake_example'] to the setup() call.
3. Edit the CmakeLists.txt to refer to cmake_example/src instead of src.

This will compile all modules into cmake_example/*.so and a "python3 setup.py install" will install those modules along with any Python files in cmake_examples/.

[loriab]

I find it a lot easier to teach CMake about installing Python files than setup.py about installing C++ modules. A first question is do you want to adopt the Linux install scheme of everything for project behind an install prefix (with subdirs bin, include, lib) or the python install scheme of everything for a project behind the current python's site-packages/mymodule?

[molpopgen]

I find it a lot easier to teach CMake about installing Python files than setup.py about installing C++ modules. A first question is do you want to adopt the Linux install scheme of everything for project behind an install prefix (with subdirs bin, include, lib) or the python install scheme of everything for a project behind the current python's site-packages/mymodule?

This is a good point. My solution is about the latter. I've modifed the cmake_example to do what I describe above. See here.

[KerstinKeller]

I very certainly need to use CMake to do the C++ build, just using a regular Extension won't be enough.

A first question is do you want to adopt the Linux install scheme of everything for project behind an install prefix (with subdirs bin, include, lib) or the python install scheme of everything for a project behind the current python's site-packages/mymodule?

This is exactly the question. My project compiles into a few .so files (pure C++), then the Python wrapper (.so) and the pure Python code. So I don't know if the "pure" C++ .so files should be installed to the system, and only the extensions to the site-packages/mymodule folder.

Right now I am following the the calling Python from CMake approach. All pure C++ modules will be installed to the system directory, the python files and compiled python extensions are first located inside my cmake build folder, and a call to setuptools on CMake install installs it to the site-packages folder.

But it also seems that I am loosing a lot of "flexibility", e.g. the possibility to let's say install my module into a virtual environment. It just feels unnatural for the way you're usually installing Python modules. (Though I guess CMake functionality could be added, too allow also for virtual environments).

The CMake from Python approach seems more natural, but it seems very hard to build everything from scratch (project includes also many third party applications). Maybe one could assume that the core part of the library (and all its dependencies) are already installed in the System, and then use Python to trigger CMake to build not the entire library but only the Python extensions. Then I would assume, that all compiled .so files (also pure C++) should end up in the site-packages folder.

[loriab]

I’d expect the separated approach (py to site-packages, cxx to system or prefix) to be difficult to manage. Whichever way you go, you want cmake to construct (possibly through configure_file()) a init.py that “from . import compiled” and “from . import interpreted” so that your pb11 shared object and any supporting python are alongside. Then no matter where installed, it’s just a matter of adding the dir of outerdir to pythonpath to envvar.

Sent from my iPhone

On Jun 20, 2018, at 6:52 AM, KerstinKeller notifications@github.com wrote:

I very certainly need to use CMake to do the C++ build, just using a regular Extension won't be enough.

A first question is do you want to adopt the Linux install scheme of everything for project behind an install prefix (with subdirs bin, include, lib) or the python install scheme of everything for a project behind the current python's site-packages/mymodule?

This is exactly the question. My project compiles into a few .so files (pure C++), then the Python wrapper (.so) and the pure Python code. So I don't know if the "pure" C++ .so files should be installed to the system, and only the extensions to the site-packages/mymodule folder.

Right now I am following the the calling Python from CMake approach. All pure C++ modules will be installed to the system directory, the python files and compiled python extensions are first located inside my cmake build folder, and a call to setuptools on CMake install installs it to the site-packages folder.

But it also seems that I am loosing a lot of "flexibility", e.g. the possibility to let's say install my module into a virtual environment. It just feels unnatural for the way you're usually installing Python modules. (Though I guess CMake functionality could be added, too allow also for virtual environments).

The CMake from Python approach seems more natural, but it seems very hard to build everything from scratch (project includes also many third party applications). Maybe one could assume that the core part of the library (and all its dependencies) are already installed in the System, and then use Python to trigger CMake to build not the entire library but only the Python extensions. Then I would assume, that all compiled .so files (also pure C++) should end up in the site-packages folder.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[yesint]

I'm also using CMake to install both C++ and Python parts. It is rather straightforward to teach CMake how to install your python files in correct places using ${PYTHON_SITE_PACKAGES}. I'm not working with Windows but on Linux it works very well. However, in general case this should depend on how much of your project is written in C++. If it mostly Python than Python-centric way may be preferable. For me it is 95% C++, so it is logical to go from C++ side.

[rfleschenberg]

@KerstinKeller From a Python developer's point of view, virtualenv is the de-facto standard for developing Python projects. So any Python library that can not be installed into a virtualenv (with pip) is a pain point.

Also, in general, I do not want to install anything that is not managed by my OS's package manager (e.g. apt) into system-wide directories. Breaking this rule is an absolute no-go for Python packages (I have seen too many systems break in almost-impossible-to-debug ways because of this). I think it is also not a great idea for C / C++ libraries.

BTW: Have you considered https://github.com/pypa/manylinux?

[spflueger]

I have a similar problem. However in my case the python modules are optional, so I definitely have to build it via cmake. Installing the python modules would be in a second step via setuptools. I also want to use the python modules directly from source when I'm developing (to avoid build/install after a python code modification). However the pybind11 library kind of messes up things here. What do you guys think about copying the built pybind11 library into the python source code directory? That seems a bit ugly to me but would solve all my problems :)

[sizmailov]

I was able to package an extension to PYPI using CMake+setuptools (link). pip triggers module compilation and installs all .so into lib/<python>/site-packages. Works smoothly for virtualenv and --user installation. The only trick is -DCMAKE_INSTALL_RPATH=$ORIGIN option. Without it python module will be not able to find other .so files.

As a python user I would appreciate a module which does not require more than pip install <module name>

[anntzer]

fwiw, https://github.com/matplotlib/mplcairo is a pybind11-based python package that is pip-installable, has manylinux (and Windows) wheels, and only relies on setuptools, not cmake; possibly you'll find something interesting in its setup.py.