Open Atry opened 9 months ago
The following workaround can suppress the error.
packages = [ pkgs.stdenv.cc.cc.lib ]
However, Ubuntu native executables, e.g. /usr/bin/git
, do not work with this workaround. It also breaks NixOS executables if they require a version of glibc or libstdc++ different from the project's.
I think the best fix for this issue is to create a wrapper of python
to set LD_LIBRARY_PATH
for Python only and support manylinux ABI out-of-box, instead of setting a project-wise LD_LIBRARY_PATH
.
LD_LIBRARY_PATH
wrapper would break subprocesses created by python.
I created a pull request helping resolving this issue.
I have created a test case here: https://github.com/cachix/devenv/blob/python-rewrite/tests/python-native-libs/devenv.nix
However for some reason it doesn't work still.
❯ python -c "import os;print(os.environ['LD_LIBRARY_PATH'])"
/home/domen/dev/cachix/devenv/.devenv/profile/lib
❯ ls .devenv/profile/lib|grep stdc++
libstdc++.so.6
❯ python -c "from ctypes.util import find_library;print(find_library('stdc++'))"
None
❯ python -c "from ctypes.util import find_library;print(find_library('GL'))"
libGL.so.1
I think LD_LIBRARY_PATH
is fragile. Maybe we should not add devenv.packages
to LD_LIBRARY_PATH
by default? To me, devenv.packages
stands for cli tools.
How about languages.python.nativeDependencies
?
I'm thinking about languages.python.libraries = config.packages
, so that it matches how other languages work, by picking it up from packages
.
That allows anyone with an issue to go and explicitly set it if needed.
If it turns out to be a bad idea, we can remove the default and issue a warning.
Do you mean you would not put config.packages
to the shell's LD_LIBRARY_PATH
?
Exactly, only pass it to python by default, but can be changed.
Sounds good to me
Sorry to bump an old thread: this problem persists (also because none of the issue or PR related has been closed or merged).
I made a few trials, and it seems like the workaround above is incompatible with an arbitrary version of Python from cachix/nixpkgs-python
.
The error is:
ImportError: /nix/store/flf14c3ibr83jsa070j25hg5gjapydhl-glibc-2.37-8/lib/libc.so.6: version `GLIBC_2.38' not found (required by /nix/store/6zl9iqg8hacf5y0h7b4agc5ym0yxzwa8-devenv-profile/lib/libstdc++.so.6)
I guess that version of Python to be using a different glibc version from the one in the stdenv
(since it's clearly fishy that glibc-2.37-8
goes looking for glibc 2.38).
Any advice about how to solve this situation?
@AleCandido Have you tried inputs.nixpkgs-python.inputs.nixpkgs.follows = "nixpkgs";
Thanks @Atry, I obviously didn't.
It recompiled Python and stopped looking for the wrong version of glibc
.
I just realized there is already a way to specify fallback library path in glibc's dynamic linker: https://github.com/NixOS/nixpkgs/pull/248547#issuecomment-1995469926
I wonder if we could use LD_AUDIT instead of LD_LIBRARY_PATH to support manylinux ABI?
Here is a demo to solve this issue using LD_AUDIT
: https://github.com/Atry/devenv-ld-floxlib
You can compare the solution with https://github.com/Atry/nix-ld-so-cache-test
Oh, that's really cool! Would that solve all glibc version incompatibility vues?
Yes. glibc should be solved by python’s RUN_PATH, not ld-floxlib
libstdc++ would be solved by ld-floxlib’s stdenv. Other shared libraries would be solved by FLOX_ENV
Here is a demo to solve this issue using LD_AUDIT: https://github.com/Atry/devenv-ld-floxlib
You're fast! It would also be nice to provide ld-floxlib in Nixpkgs. Also we could patch it to reproduce the LD_FALLBACK_PATH
interface too
~I wish someone wrote this in Rust, to be sure it's safe.~ Not possible due to dynamic linker restrictions
@Atry can you make a PR with a proper implementation using floxlib?
@domenkozar Would you mind reviewing other PRs first? https://github.com/cachix/devenv/pull/1018
can you make a PR with a proper implementation using floxlib?
Just to help with this. My understanding is a general fix is basically changing the python derivation to be like @ Atry's existing flake, plus
If that doesnt sound far off from reality, I might try it a month from now
~I wish someone wrote this in Rust, to be sure it's safe.~ Not possible due to dynamic linker restrictions
@domenkozar can you elaborate on why this isn't possible in Rust? I was planning on giving it a try sometime soon ;).
I don't know why but I tried to create a LD_AUDIT hook using statically linked musl instead of glibc and it always crashes.
On Tue, Apr 30, 2024 at 2:03 AM Sem Mulder @.***> wrote:
I wish someone wrote this in Rust, to be sure it's safe. Not possible due to dynamic linker restrictions
@domenkozar https://github.com/domenkozar can you elaborate on why this isn't possible in Rust? I was planning on giving it a try sometime soon ;).
— Reply to this email directly, view it on GitHub https://github.com/cachix/devenv/issues/773#issuecomment-2084771646, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAES3ORWHIUFT54LIWQXKUDY75M6DAVCNFSM6AAAAAA3E7ZW36VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBUG43TCNRUGY . You are receiving this because you were mentioned.Message ID: @.***>
I don't know why but I tried to create a LD_AUDIT hook using statically linked musl instead of glibc and it always crashes.
Might be because musl
and glibc
are not binary compatible? Per https://www.musl-libc.org/faq.html:
Binary compatibility is much more limited, but it will steadily increase with new versions of musl. At present, some glibc-linked shared libraries can be loaded with musl, but all but the simplest glibc-linked applications will fail if musl is dropped-in in place of /lib/ld-linux.so.2.
I used to think that a statically linked musl can coexist with a dynamically linked glibc.
On Tue, Apr 30, 2024 at 2:56 AM Sem Mulder @.***> wrote:
I don't know why but I tried to create a LD_AUDIT hook using statically linked musl instead of glibc and it always crashes.
Might be because musl and glibc are not binary compatible? Per https://www.musl-libc.org/faq.html:
Binary compatibility is much more limited, but it will steadily increase with new versions of musl. At present, some glibc-linked shared libraries can be loaded with musl, but all but the simplest glibc-linked applications will fail if musl is dropped-in in place of /lib/ld-linux.so.2.
— Reply to this email directly, view it on GitHub https://github.com/cachix/devenv/issues/773#issuecomment-2084871280, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAES3OQBORWJ2UYSGZUEGG3Y75TFFAVCNFSM6AAAAAA3E7ZW36VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBUHA3TCMRYGA . You are receiving this because you were mentioned.Message ID: @.***>
I can't say if the ABI would cause problems, but I can say a statically linked musl with a dynamically linked glibc sounds like a cursed runtime
Describe the bug Python packages installed from
pip
cannot findlibstdc++.so.6
To reproduce Given the
flake.nix
Run
nix develop --impure
, then I got the errorVersion
e91205acb792f6a49ea53ab04c04fbc2a85039cd
This error can be reproduced on both Ubuntu and NixOS