[python] ctypes.util.find_library should return full path to the library

[kirelagin]

I’m trying to run a Python project that uses libnfc in nix-shell. The code does

ctypes.util.find_library('nfc')

and for some strange reason this returns just libnfc.so.5. I’m not sure how find_library works, but the docs say:

On Linux, find_library() tries to run external programs (/sbin/ldconfig, gcc, and objdump) to find the library file.

Unfortunately this doesn’t seem to work, even though g++ successfully builds a C++ program that depends on libnfc. Any ideas why this is happening? Why isn’t it able to figure out the path to libnfc? Shouldn’t ctypes.util.find_library be patched in nixpkgs to avoid running weird programs and just check the environment variables?

[domenkozar]

Can you provide exact steps to reproduce?

[kirelagin]

# shell.nix
let pkgs = import <nixpkgs> {};
in with pkgs;
stdenv.mkDerivation rec {
  name = "test";

  src = ./.;

  buildInputs = [
    python2
    libnfc
  ];
}

# nfc.py
import ctypes
import ctypes.util

lib = ctypes.util.find_library('nfc')
print('Library: ' + lib)

ctypes.CDLL(lib)

$ python2 nfc.py
Library: libnfc.so.5
Traceback (most recent call last):
  File "nfc.py", line 7, in <module>
    ctypes.CDLL(lib)
  File "/nix/store/5aq9sjd0g4frmpqni0a6zykzkcnzc3al-python-2.7.9/lib/python2.7/ctypes/__init__.py", line 365, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libnfc.so.5: cannot open shared object file: No such file or directory

// nfc.c 
#include <nfc/nfc.h>

int main()
{
    nfc_context *c;

    nfc_init(&c);
    nfc_exit(c);

    return 0;
}

$ gcc -lnfc nfc.c
# OK

[kirelagin]

I assumed this was expected to work, because I saw this patch. But now I looked around nixpkgs and seems that everyone else is patching find_library calls with fixed paths.

I also reread the docs and noticed, it says that find_library returns just the library file name, not the path on Linux (unlike other OS). So, hm, probably the question is then that the loader has to be patched…

[domenkozar]

@kirelagin for now set DYLD_LIBRARY_PATH to point to ${libnfc}/lib will do, but we should find_library to return fullpath.

[domenkozar]

It's a bit harder since ldconfig -p fails on NixOS since we don't keep the cache.

[kirelagin]

but we should find_library to return fullpath.

As I said, Python docs state that on Linux find_library returns just the filename, so patching it this way might break things…

[domenkozar]

It can't break things more than they currently are - as it doesn't even work due to prefix being unknown.

[domenkozar]

Yeah I'm getting bitten by this one again.

[domenkozar]

This is the line that gets executed on linux: https://github.com/python/cpython/blob/2.7/Lib/ctypes/util.py#L242

[domenkozar]

So if you were to execute ctypes.util.find_library('c') it would do:

1. LANG=C LC_ALL=C gcc -Wl,-t -o /tmp/tempfile>&1 -l c and store the output
2. objdump -p -j .dynamic objdump -p -j .dynamic /nix/store/hd6km3hscbgl2yw8nx7lr5z9s8h89p04-glibc-2.21/lib/libc.so.6|grep SONAME

I think we should replace this with our own implementation

[lucabrunox]

I believe all the code using find_library without an absolute path is broken, regardless. However for c it might be hardcoded to the nix path.

[domenkozar]

That's the thing, I don't want to maintain every package and patch it when it uses find_library

[lucabrunox]

@domenkozar well, I don't think there's any good solution to that unfortunately.

[lucabrunox]

@domenkozar perhaps a special env variable PY_CTYPES_LIBPATH, to look for libs inside that path. Similar to LD_LIBRARY_PATH, but not LD_LIBRARY_PATH.

[domenkozar]

We could use http://linux.die.net/man/8/ldconfig to generate the cache at build time. find_library then uses ldconfig -p to get the available paths.

[Profpatsch]

(triage) there have been a few patches, is this fixed now?

[domenkozar]

We haven't solved it universally yet.

[FRidh]

perhaps a special env variable PY_CTYPES_LIBPATH, to look for libs inside that path. Similar to LD_LIBRARY_PATH, but not LD_LIBRARY_PATH.

We could use http://linux.die.net/man/8/ldconfig to generate the cache at build time. find_library then uses ldconfig -p to get the available paths.

Both are interesting options.

dlopen is called during runtime. If we somehow want to hardcode the paths, we need to make sure all calls to dlopen are executed during the build. Otherwise, the libraries need to be found during runtime by the interpreter or whichever application is using the Python library.

So, if we consider the ldconfig cache, then the cache has to be available during runtime, and the interpreter has to be able to find it. As soon as we put packages together in say an env we end up with multiple caches, so we would have to rebuild the cache as well.

We should be able to patch the interpreter to load an environment variable that points to the cache. Then we can build the cache 1) whenever we build a Python package and 2) when we build an env.

A downside is that when building a cache all buildInputs would become runtime dependencies because they're listed in the cache. Furthermore, ldconfig in python.buildEnv is only passed a list of caches. Therefore, we might also need an environment variable NIX_PY_CTYPES_LIBPATH which files are added to a ld.so.conf file that is then used by ldconfig.

So, we patch Python to check for NIX_PY_CTYPES_LIBPATH and then buildPythonPackage and python.buildEnv

1. check for NIX_PY_CTYPES_LIBPATH. If it exists, create a /nix/ld.so.conf file
2. if /nix/ld.so.conf exists, build the cache ldconfig -CNIX_PY_LDCONFIG_CACHE -f /nix/ld.so.conf, and setNIX_PY_LDCONFIG_CACHE=$out/nix/nix-py-ldconfig-cache;Note that/nix/ld.so.confof dependencies should also be taken into account, so maybe its just better to add these libraries topropagatedBuildInputs`.
3. Now the interpreter can find the cache during build time.
4. For runtime we support only python.buildEnv where we rebuild the cache and again set NIX_PY_CTYPES_LIBPATH

[mmahut]

Are there any updates to this issue, please?

[stale[bot]]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.

[toyo-chi]

Trying to build another Python library and get the same error. Looks like it's still important.

[FRidh]

A downside is that when building a cache all buildInputs would become runtime dependencies because they're listed in the cache. Furthermore, ldconfig in python.buildEnv is only passed a list of caches.

This is not actually an issue. buildInputs represents the runtime deps. Other build-time deps should be in nativeBuildInputs anyway.

[michaelcadilhac]

Still unsure what's a valid workaround here. Setting DYLD_LIBRARY_PATH seems to have no effect.

[fluffynukeit]

I am also trying to fix the a find_library call in a python package, for libsnd. Like @michaelcadilhac , setting DYLD_LIBRARY_PATH has no effect.

in pkgs.mkShell {
  buildInputs = [ 
    myenv
    pkgs.ffmpeg
    pkgs.libsndfile
  ];
  DYLD_LIBRARY_PATH="${pkgs.libsndfile}/lib";
}

DYLD_LIBRARY_PATH=/nix/store/pbx64k731w2r2501y500r8n5fxy6m91r-libsndfile-1.0.30-bin/lib

OSError: cannot load library 'libsndfile.so.1': libsndfile.so.1: cannot open shared object file: No such file or directory

Edit: I am using mach-nix, and the problem was that I had configured mach-nix (as default) to prioritize wheel and sdist packages. Instead, the nixpkgs version of pysoundfile already handles patching to find libsndfile. I just had to add this to my mach-nix.mkPython arguments: providers.soundfile = "nixpkgs";

I am still running into the problem for other python packages using dlopen, though, specifically ones utilizing opencl.

[stale[bot]]

I marked this as stale due to inactivity. → More info

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/screenshot-with-mss-in-python-says-no-x11-library/14534/4

[stale[bot]]

I marked this as stale due to inactivity. → More info

[wilhelmy]

Just for clarification, it seems possible to fix this issue by patching ctypes.util.find_library to return the full path? Should we contact upstream about possible side-effects?

[reivilibre]

I marked this as stale due to inactivity. → More info

I'm still interested in this issue. It's just bitten me now and I worry I'm not experienced enough to solve this properly.

DYLD_LIBRARY_PATH did not work for me either.

[wilhelmy]

@reivilibre Have you tried the workaround (patching the call out)? https://discourse.nixos.org/t/screenshot-with-mss-in-python-says-no-x11-library/14534/4

[rkjnsn]

I'm running into this issue with a Calibre plugin. Even if I add the required library to Calibre's deps, the plugin, which uses find_library, can't find it. Because the actual code is part of a plugin installed in the my home directory, not by nix, patching it to use an absolute path instead isn't really practical, and would break every time the path changed due to an update+garbage-collect.

[domenkozar]

I've went really deep into this one after all this time, so I'll document my findings.

I've pushed the changes into https://github.com/cachix/devenv/pull/745, but the crucial bit is here:

https://github.com/cachix/devenv/blob/7bbcb9f53a6c8e0ced952feeb1a01ccff6e362cb/src/modules/languages/python.nix#L13

We wrap LD_LIBRARY_PATH around python interpreter and give it an explicit list of packages it will be able to link from.

This relies on https://github.com/python/cpython/commit/82df3b3071bb003247c33eac4670775e9883c994, which is only present in Python 3.6 or higher. (note for older interpreter I've opened https://github.com/cachix/nixpkgs-python/issues/17 that I plan to address one day).

I've added support for manylinux too, but libraries like stdc++ still can't be loaded:

ld -t -L $(nix-build -A pythonManylinuxPackages.manylinux2014Package)/lib -lstdc++
/nix/store/zkjq96ik8cbv6ijh1lylnkk2bni9qvas-binutils-2.40/bin/ld: cannot find -lstdc++: No such file or directory

ld -t -L $(nix-build -A stdenv.cc.cc.lib)/lib -lstdc++
/nix/store/c50v7bf341jsza0n07784yvzp5fzjpn5-gcc-12.3.0-lib/lib/libstdc++.so

If I name it libstdc++.so instead of libstdc++.so.6 in manylinux wheel it works, but I don't understand what's going on.

Last but not least, for venv to work we need to merge https://github.com/NixOS/nixpkgs/pull/250935 to pick up the right interpreter.

Using all this the following devenv.nix works (on a branch I have, to be 1.0):

{ pkgs, ... }: {
  packages = [ pkgs.cairo ];

  languages.python = {
    enable = true;
    venv.enable = true;
    venv.requirements = ''
      pillow
    '';
  };
}

And test it using devenv shell python -c "from PIL import Image".

[SomeoneSerge]

Shouldn’t ctypes.util.find_library be patched in nixpkgs to avoid running weird programs and just check the environment variables?

That's a possibility, but the issue is that there is no correct behaviour for ctypes.util.find_library at all. The upstream implementation might return build or host platform's dependencies at random, and in both cases the function relies on glibc and gcc internals. The most common practice seems to be to use it for locating native dependencies (i.e. what'd be detected by ld.so or ldconfig, as opposed to what'd be detected by gcc), but that's just us making assumptions. So which kind should we return?

If we were to patch ctypes... what about just introducing a separate environment variable to list search paths for find_library specifically? We could keep the original implementation as a fallback too, and issue UserWarnings whenever those branches are reached