Bundle ZMQ in manylinux wheels?

[alexrudy]

In my conintuing silliness of mixing pyzmq and libzmq via Cython, I've noticed that when the manylinux wheel is installed, it effectively bundles the libzmq library, in zmq/.libs/, which makes it difficult to compile other python extensions linked against the libzmq in use by pyzmq. (This is done by the manylinux wheel machinery)

Explicitly bundling libzmq when building manylinux would make it much easier to link against pyzmq's copy of libzmq. Otherwise I just force pyzmq to never use binaries, but this seems silly when the solution is already built into pyzmq.

[minrk]

I think it's much better to use the 'real' libzmq build system rather than bundling libzmq where possible, so I'd rather not move to bundled builds on Linux. Instead, I'd prefer to solve the problem by pulling in zmq.h in the linux wheels even when not bundling. How does that sound?

[alexrudy]

Using the "real" build system makes sense, and I can see why you want to keep it that way.

The problem isn't so much zmq.h, because we could bundle that.

Rather, its that the many linux wheel munges the library name and hides it away. For example, in pyzmq-16.0.2-cp27-cp27mu-manylinux1_i686.whl, the library is at zmq/.libs/libzmq-c3d2772d.so.5.0.2, but in pyzmq-16.0.2-cp27-cp27mu-manylinux1_x86_64.whl the library is at zmq/.libs/libzmq-0da13fcf.so.5.0.2.

Not only is it a pain to chase around the changing library name, but it is very difficult to specify this library name via distutils.

I guess I could always glob for the full name with the hash appended, and link against that. But that requires changing any build script which assumes the library is named libzmq.*. This means that I have to make build scripts specifically detect this case, when really zmq.get_library_dirs should be enough. Would including zmq.get_library_name in the core module be ok?

[minrk]

I didn't think of the name issue. Adding get_library_name makes sense.

[alexrudy]

I'm thinking a little harder, and I think it makes more sense in my case to expose the compiler settings more as a single method. I think this can be mostly done by loading compiler.json from the source tree, but with a few modifications to account for the moved library in manylinux. I'll work on something to do that.

[njsmith]

Note that the library name is based on a hash of the libzmq.so file's contents, so new pyzmq releases will likely have different names. This could create a problem in the situation where you install pyzmq, build your package with -lzmq-XXXX, and then pip install -U pyzmq and now it's called libzmq-YYYY.

The purpose of the hashing is to avoid accidental collisions between packages -- if foo.whl bundles libzmq.so.5.1, bar.whl bundles libzmq.so.5.2, and maybe you have a locally compiled package using the system's libzmq.so.5.0, and then you import all those packages into the same Python, you may end up having a very bad time when ld.so picks one of those libzmq.so's at random and then uses it for all three packages.

You could tweak your wheels so that they use a mangled name like libzmq-for-pyzmq-wheel.so or something, that's unique but stable across pyzmq releases. (This would need some hacking on auditwheel or whatever, but it's just a python script.) Though, note that you're then effectively adding your bundled libzmq's ABI to your public API, which has consequences you'll probably want to think through.

Also, this simple hack is linux specific -- in particular exporting the bundled libzmq from macos wheels is extraordinarily difficult.