Better support for python packaging and virtualenvs

I'm working on libtorrent's python bindings and I'm finding that it's really difficult to package and test b2 python targets using python ecosystem standards.

Python extensions are normally built by distutils, which always builds them against the running python interpreter. That is, if you build using python3.6 setup.py bdist_wheel, you'll get a wheel build against the python3.6 you ran.

This dovetails with the standard python package testing workflow, which looks like this:

Set up a virtual environment
Build the package in that environment, for the environment
Install the package
Run tests against the installed code

Note that distutils' bdist_wheel is actually the only standard tooling to create python wheels. So if you want to package a python target from b2 into a wheel, the most straightforward way is to write a setup.py, and customize its build_ext step to invoke b2 instead of building using distutils logic.

So for both packaging and testing, we need to get b2 to follow the python practice of build for a given interpreter. But b2's design makes this hard.

We can try b2 python=X.Y, but we need to ensure this matches some using python : X.Y : ... in some *-config.jam, which matches our given environment and not others, and isn't overridden by any other config.

The problem is that b2 treats python like gcc, as some tool installed globally by a supervisor in one or more versions. But this is not how python works. Local, temporary virtual environments are the rule, not the exception. We certainly can't rely on ~/user-config.jam to have what we need.

In libtorrent, I ended up having setup.py create a temporary project-config.jam like this:

import feature ;
feature.feature libtorrent-python : on ;
using python : X.Y : /path/to/my/pythonX.Y : ... : ... : <libtorrent-python>on : ... ;

Then, setup.py invokes b2 python=X.Y libtorrent-python=on .... The dummy libtorrent-python feature lets me select only my dynamically-created python configuration.

We're only halfway done, because we can't trust python.jam's other configuration guesses.

I believe the code that guesses include and library search paths has never been tested in virtual environments. It doesn't honor the difference between sys.exec_prefix and sys.base_exec_prefix.

Worse than just being wrong on some platforms, the guesses appear kind of willy-nilly. For instance, it applies a "pythonX.Y/config" library search path on all non-windows platforms, but this appears to only be appropriate for cygwin.

Worse, the python config rule only allows a single library search path. There are certainly cases where we need multiple ones. I ended up resolving this by adding extra library-path=... requirements to my b2 command. It's not really appropriate to apply these globally, but I couldn't think of another solution.

distutils also has code that guesses include and library paths, and it does a much better job. For our autogenerated python config, we override include and library paths with the include_dirs and library_dirs attributes of distutils.command.build_ext.build_ext. In case these are empty, I do not trust python.jam's guesses not to be destructive, and I configure garbage search paths instead.

Finally, we want our build_ext to produce an artifact with name and location it normally would, so other distutils logic can consume it. python.jam is unhelpful again here, as its configuration guesses aren't helpful and it appends a platform-specific suffix to whatever was configured. So if we get a correct value from distutils, we must anticipate python.jam's mangling and de-mangle our value correctly.

I think in the short term:

I should be able to invoke b2 python.exe=/path/to/my/pythonX.Y
- This should not require, and should disregard, any config in *-config.jam
- This should invoke the executable to discover include and library search paths, and configure the extension suffix, using logic directly from build_ext within distutils
python.jam's existing library and include search path guesses should be correct for virtual environments, on all platforms

In the long term, I think the boost team should come up with a more clear story for how a user can use b2 python targets with standard python tools, and package them in standard ways. This was way too much work.

boostorg / build

Better support for python packaging and virtualenvs #706