Mixing system packages with python results in shared library errors

[bobvanderlinden]

Describe the bug

When using system packages from within a python application symbol lookup error may happen. This happens when the system package (like git) is linked against a different version of libc compared to the python version (from devenv) is linked against.

A specific case where this happens more often occurs when using a remote git repository as a requirement for your devenv-based python project. devenv calls pip install -r requirements.txt, which will run git. git is usually not defined inside devenv, thus the required libc is likely to be different from what is used in devenv. The python wrapper within devenv runs python with LD_LIBRARY_PATH=$DEVENV_PROFILE/lib, which will force ld to use libraries within $DEVENV_PROFILE/lib. The libc that resides in $DEVENV_PROFILE/lib is a different version than what git assumes to be ld-ed against.

This results in errors like the following:

Collecting git+https://github.com/pypa/sampleproject.git (from -r /nix/store/g41hqm03k2179v5pkyv889gyrgb370g0-requirements.txt (line 1))
  Cloning https://github.com/pypa/sampleproject.git to /tmp/nix-shell.07Ag0j/nix-shell.tKP4lT/pip-req-build-jidkerj5
  error: subprocess-exited-with-error

  × git version did not run successfully.
  │ exit code: 127
  ╰─> [2 lines of output]
      git: error while loading shared libraries: __vdso_gettimeofday: invalid mode for dlopen(): Invalid argument
      [end of output]

Potential solutions

• Adding git to packages in devenv will resolve the problem with git, but on my system I got the next binary that resulted in the a similar error (gh). Adding all personal tools is one way to get over this problem.
• Avoid using other binaries within the python application by not using git.
• Resetting LD_LIBRARY_PATH upon calling other binaries from within the wrapped devenv python.
• Make sure Python libraries are always linked against specific (working) versions of native libraries. This can be done using patchelf/autopatchelf. In addition, LD_LIBRARY_PATH should not be used so that ld won't be forced to use different libraries. See PR https://github.com/cachix/devenv/pull/840 as a poc.

To reproduce

Follow the instructions from the readme:

https://gist.github.com/bobvanderlinden/f31299079b67ddcc1b2c6029cf6569f6

Version

devenv 1.0.2 (x86_64-linux)

[shoskensMagics]

Also running into this when calling system commands from within my Python application. Current band-aid solution is to reset LD_LIBRARY_PATH before doing any system command.

[ddogfoodd]

@shoskensMagics you mean running unset LD_LIBRARY_PATH?

[bobvanderlinden]

It's a bit tricky to workaround this. unset LD_LIBRARY_PATH wouldn't really 'just work'. devenv adds LD_LIBRARY_PATH in the wrapper for python. I don't know of a way (currently) to avoid devenv from doing this when languages.python.enable = true. Inside the python program you'd have to del os.environ['LD_LIBRARY_PATH'] to avoid it to propagate to child-processes.

I made https://github.com/cachix/devenv/pull/1542, so that we can control the python-wrapper package (and force devenv to use python as-is without wrappers).

---

Apart from the workaround, I also found LD_LIBRARY_PATH to sometimes break things when debugging python inside vscode. The terminal is wrapped with some python tool, which will fail to start the terminal (I think because it depends on an unexpected libc)

[vlaci]

What I've done is to set-up a pth hack to circumvent this issue:

  tasks = {
    "venv:libraryPath" = {
      exec = ''
        # Subprocesses executed from python shouldn't inherit devenv's library path hack
        SITE_PACKAGES=($VIRTUAL_ENV/lib/python3*/site-packages)
        echo 'import os; os.environ.pop("LD_LIBRARY_PATH", None)' > $SITE_PACKAGES/devenv.pth
      '';
      after = [ "devenv:python:poetry" ];
      before = [ "devenv:enterShell" ];
    };

this works, because all .pth files are executed as python code during interpreter startup, if they start with an import statement