BUG: error: cannot find library libsodium-37cdab92.so.26.1.0

[rayl-bln]

This is a pyzmq bug

• [X] This is a pyzmq-specific bug, not an issue of zmq socket behavior. Don't worry if you're not sure! We'll figure it out together.

What pyzmq version?

26.1.0

What libzmq version?

4.3.5

Python version (and how it was installed)

Python 3.12.3, ubuntu 24.04

OS

ubuntu 24.04

What happened?

• Developing a python 3.12 program using pyzmq
• Using poetry to manage dependencies
• Everything works fine inside a docker container

When I try to package it using dh-virtualenv and dh-poetry, I get a failure on the "dh_shlibdeps" step.

dpkg-shlibdeps: error: cannot find library libsodium-37cdab92.so.26.1.0 needed by debian/xxx/opt/xxx/xxx/lib/python3.12/site-packages/pyzmq.libs/libzmq-4192fc97.so.5.2.5 (ELF format: 'elf64-x86-64' abi: 'ELF:64:l:amd64:0'; RPATH: '')

the library seems to be properly included in pyzmq-26.1.0-cp312-cp312-manylinux_2_28_x86_64.whl

root@1a867745cb91:/opt/src# ls -l debian/xxx/opt/xxx/xxx/lib/python3.12/site-packages/pyzmq.libs/
total 2124
-rw-r--r-- 1 root root 1221032 Aug  9 23:54 libsodium-37cdab92.so.26.1.0
-rw-r--r-- 1 root root  948800 Aug  9 23:54 libzmq-4192fc97.so.5.2.5

Code to reproduce bug

No response

Traceback, if applicable

no traceback, just a build error

More info

No response

[minrk]

Sounds like this should be reported to dh-virtualenv, not here. If the lib is there and it's found at import, something is wrong is dh-virtualenv's shlib search.

If it helps, this wheel is built with the now-standard auditwheel/repairwheel workflow used by many wheels, manylinux, cibuildwheel, etc.

[rayl-bln]

thanks, i've opened https://github.com/spotify/dh-virtualenv/issues/365

i do wonder if the lib would always be found at import.

a simple 'ldd' on the libzmq shared object can't find libsodium either.

(.venv) ubuntu@ce5ac35e10ed:/opt/src$ ldd .venv/lib/python3.12/site-packages/pyzmq.libs/libzmq-4192fc97.so.5.2.5
        libsodium-37cdab92.so.26.1.0 => not found

this is the only out of 316 deps across 261 shared objects in my virtualenv that does not resolve.

find .venv/ -name \*.so -o -name \*.so.\* | while read f; do echo "============ $f"; ldd $f; done

note that this shared object DOES find the libsodium dep

(.venv) ubuntu@ce5ac35e10ed:/opt/src$ ldd .venv/lib/python3.12/site-packages/zmq/backend/cython/_zmq.cpython-312-x86_64-linux-gnu.so
        libzmq-4192fc97.so.5.2.5 => /opt/src/.venv/lib/python3.12/site-packages/zmq/backend/cython/../../../pyzmq.libs/libzmq-4192fc97.so.5.2.5 (0x00007f1498542000)
        libsodium-37cdab92.so.26.1.0 => /opt/src/.venv/lib/python3.12/site-packages/zmq/backend/cython/../../../pyzmq.libs/libsodium-37cdab92.so.26.1.0 (0x00007f14981fc000)

it been years since i last looked at this stuff, but IIRC this might have something to do with RPATH?

(.venv) ubuntu@ce5ac35e10ed:/opt/src$ objdump -p .venv/lib/python3.12/site-packages/pyzmq.libs/libzmq-4192fc97.so.5.2.5 | grep RPATH
(.venv) ubuntu@ce5ac35e10ed:/opt/src$ objdump -p .venv/lib/python3.12/site-packages/zmq/backend/cython/_zmq.cpython-312-x86_64-linux-gnu.so | grep RPATH
  RPATH                $ORIGIN/../../../pyzmq.libs

could it be possible that the libzmq object needs that same RPATH set during wheel build, so that it can also find the libsodium dep?

[minrk]

Can you import zmq in the env? If not, that's definitely a pyzmq problem! If it works, the detection tool is not accurately detecting the how libraries get loaded.

It's worth noting that both libzmq and libsodium are exclusively loaded by that _zmq. So maybe loading libzmq directly on its own wouldn't work, but that doesn't matter in practice because it never happens, except by this tool which makes imprecise assumptions. If that's the case, perhaps the package to report the issue to is auditwheel, because it may only be performing the rpath patch on Extension modules, but not on intermediate libs like libzmq in pyzmq, which have transitive dependencies (_zmq -> libzmq -> libsodium). And maybe there's a flag pyzmq should be passing to auditwheel.

[rayl-bln]

import zmq definitely works, so i'll chase the various tools folks. thanks for reviewing!

[minrk]

I think this is already reported in auditwheel as https://github.com/pypa/auditwheel/issues/451 . Apparently audithweel fixes RPATH when it's defined, but doesn't add it. Adding (unnecessary, but not wrong) RPATH when we build libzmq seems to solve the problem: #2014. When it finishes, can you test with the appropriate wheel from this run? I think it will work.

[rayl-bln]

Looks good! built the deb with the new wheel and did a quick sanity check, seems to work fine. thanks for the speedy fix :-)

[minrk]

Thanks for testing!