glibc-2.30 libraries link to glibc-2.27

neilmayhew commented 4 years ago

Describe the bug

The libraries in glibc-2.30 link to the libc from glibc-2.27

To Reproduce

$ ls -l $(nix-build --no-out-link -A glibc)/lib/libpthread-2.30.so
-r-xr-xr-x 1 root root 146088 Dec 31  1969 /nix/store/9rabxvqbv0vgjmydiv59wkz768b5fmbc-glibc-2.30/lib/libpthread-2.30.so
$ ldd $(nix-build --no-out-link -A glibc)/lib/libpthread-2.30.so
  linux-vdso.so.1 (0x00007fff2ffb9000)
  libc.so.6 => /nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/libc.so.6 (0x00007f7c9f4a3000)
  /nix/store/9rabxvqbv0vgjmydiv59wkz768b5fmbc-glibc-2.30/lib/ld-linux-x86-64.so.2 => /nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib64/ld-linux-x86-64.so.2 (0x00007f7c9f67c000)

Expected behavior

$ ldd $(nix-build --no-out-link -A glibc)/lib/libpthread-2.30.so
  linux-vdso.so.1 (0x00007fff2ffb9000)
  libc.so.6 => /nix/store/9rabxvqbv0vgjmydiv59wkz768b5fmbc-glibc-2.30/lib/libc.so.6 (0x00007f7c9f4a3000)
  /nix/store/9rabxvqbv0vgjmydiv59wkz768b5fmbc-glibc-2.30/lib/ld-linux-x86-64.so.2 => /nix/store/9rabxvqbv0vgjmydiv59wkz768b5fmbc-glibc-2.30/lib64/ld-linux-x86-64.so.2 (0x00007f7c9f67c000)

Additional context

This seems to be the cause of a number of other problems, in particular symbols not being found when they should be, eg, when I try to run scripts in sublime3:

bash: relocation error: /nix/store/9rabxvqbv0vgjmydiv59wkz768b5fmbc-glibc-2.30/lib/libpthread.so.0: symbol __write_nocancel version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference

The same error is reported in #79954. I think the fix for that (#80005) should be reverted because it seems like zoom-us still needs v4l. (My camera no longer works, for example.) However, I can't revert it locally because the glibc problem prevents v4l from pre-loading with zoom-us.

Metadata

system: "x86_64-linux"
host os: Linux 4.19.107, NixOS, 19.09.2346.85d879e5133 (Loris)
multi-user?: yes
sandbox: yes
version: nix-env (Nix) 2.3.3
channels(root): "nixos-19.09.2346.85d879e5133"
channels($USER): "nixpkgs-20.09pre218717.05f0934825c"
nixpkgs: /home/$USER/.nix-defexpr/channels/nixpkgs

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:

nixos-discourse commented 4 years ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/go-no-go-meeting-nixos-20-03-markhor/6495/12

Ma27 commented 4 years ago

Is it possible that this is related to the bootstrap-files? When I run nix-build -A stdenv.bootstrapTools, I get a lot of libraries linked to 2.27 in the store-path. Since our glibc is built with the stuff from the bootstrap-tools IIRC, a regeneration of all bootstrap-files would solve this, right?

infinisil commented 4 years ago

Cannot reproduce on master (cab6b019b17881f42f618dcc9fce9341f1a6df81). Trying with the following repro.nix:

let
  pkgs = import (fetchTarball {
    url = "https://github.com/NixOS/nixpkgs/tarball/cab6b019b17881f42f618dcc9fce9341f1a6df81";
    sha256 = "0gqh7yj9vlna6vrmrra75gwkwcbj5fshdp6jrw7y2s5l1aixn388";
  }) { system = "x86_64-linux"; overlays = []; config = {}; };
in pkgs.runCommandNoCC "repro-84043" {} ''
  ${pkgs.glibc.bin}/bin/ldd ${pkgs.glibc}/lib/libpthread-2.30.so > $out
''

Then

$ cat $(nix-build repro.nix)
  linux-vdso.so.1 (0x00007ffff7fd3000)
  libc.so.6 => /nix/store/9rabxvqbv0vgjmydiv59wkz768b5fmbc-glibc-2.30/lib/libc.so.6 (0x00007ffff7dee000)
  /nix/store/9rabxvqbv0vgjmydiv59wkz768b5fmbc-glibc-2.30/lib/ld-linux-x86-64.so.2 => /nix/store/9rabxvqbv0vgjmydiv59wkz768b5fmbc-glibc-2.30/lib64/ld-linux-x86-64.so.2 (0x00007ffff7fd4000)

Ma27 commented 4 years ago

Hmm.... I actually can't either (on NixOS 20.03). This may be related to the fact that @neilmayhew is on 19.09 (which actively uses glibc-2.27).

@neilmayhew at which git-rev of nixpkgs did you run ldd $(nix-build --no-out-link -A glibc)/lib/libpthread-2.30.so?

However, it's getting late... unless someone's faster, I can take a look at this tomorrow :)

neilmayhew commented 4 years ago

I was at 05f0934825c (see nix-info above).

However, I've just discovered something interesting. When I run the repro.nix above, I get the same results as @Infinisil (libc.so.6 resolves to glibc-2.30). However, when I run ldd in a regular interactive shell on exactly the same file, from the same store path, I get a different result (libc.so.6 resolves to glibc-2.27). I unset LD_LIBRARY_PATH and that didn't make a difference. I assume it's because:

$ readlink -f $(which ldd)
/nix/store/hblpx8x5w88igmwa4ydnsnb65s363lji-glibc-2.27-bin/bin/ldd

This is because ldd is coming from my system environment (ie nixos-19.09) rather than from my profile. However, this works:

$ nix-shell -p glibc
$$ which ldd
/nix/store/c2lx3lsqp0y5kwvwkqma8iihc7xjj1m8-glibc-2.30-bin/bin/ldd
$$ ldd /nix/store/9rabxvqbv0vgjmydiv59wkz768b5fmbc-glibc-2.30/lib/libpthread-2.30.so 
  linux-vdso.so.1 (0x00007fffedbd8000)
  libc.so.6 => /nix/store/9rabxvqbv0vgjmydiv59wkz768b5fmbc-glibc-2.30/lib/libc.so.6 (0x00007f9344423000)
  /nix/store/9rabxvqbv0vgjmydiv59wkz768b5fmbc-glibc-2.30/lib/ld-linux-x86-64.so.2 => /nix/store/9rabxvqbv0vgjmydiv59wkz768b5fmbc-glibc-2.30/lib64/ld-linux-x86-64.so.2 (0x00007f9344605000)

Even so, I'm surprised that the path to libc6 isn't enforced by libpthread-2.30.so. In particular,

patchelf --print-rpath /nix/store/9rabxvqbv0vgjmydiv59wkz768b5fmbc-glibc-2.30/lib/libpthread-2.30.so
(empty output)

So there is no rpath. This is confirmed by running readelf -d. Is this normal for glibc libs?

Ma27 commented 4 years ago

So, this seems to be depending on the environment.

I just built a NixOS 19.09 VM (which uses glibc-2.27) and I could reproduce the issue. I checkout out nixpkgs at 318f05f7761ded5dc0f5eab1d1787006c11b00be and built glibc like this:

[root@testvm:~/nixpkgs]# nix-build -A glibc
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
[ lots of downloads ]
/nix/store/9rabxvqbv0vgjmydiv59wkz768b5fmbc-glibc-2.30

Right now, the issue is actually reproducible:

[root@testvm:~/nixpkgs]# ldd ./result/lib/libpthread.so
    linux-vdso.so.1 (0x00007ffc7d98a000)
    libc.so.6 => /nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/libc.so.6 (0x00007f2df349e000)
    /nix/store/9rabxvqbv0vgjmydiv59wkz768b5fmbc-glibc-2.30/lib/ld-linux-x86-64.so.2 => /nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib64/ld-linux-x86-64.so.2 (0x00007f2df3677000)

However, when I change the environment with nix-shell -I nixpkgs=`pwd` -p, the issue disappears:

[nix-shell:~/nixpkgs]# ldd result/lib/libpthread.so
    linux-vdso.so.1 (0x00007ffe45771000)
    libc.so.6 => /nix/store/9rabxvqbv0vgjmydiv59wkz768b5fmbc-glibc-2.30/lib/libc.so.6 (0x00007f1d177c2000)
    /nix/store/9rabxvqbv0vgjmydiv59wkz768b5fmbc-glibc-2.30/lib/ld-linux-x86-64.so.2 => /nix/store/9rabxvqbv0vgjmydiv59wkz768b5fmbc-glibc-2.30/lib64/ld-linux-x86-64.so.2 (0x00007f1d179a4000)

This raises the following questions for me:

The issue only appears if you're on 19.09 and you try to use packages from 20.03 or unstable, right? (@domenkozar may I ask if that's the case in your setup as well?). If that's right, I guess that this appears more often due to the big version jump (I helped out shipping this update and especially 2.28 and 2.29 seemed to have relatively many changes).
Isn't this issue 'fixable" by an upgrade to 20.03? (I couldn't reproduce this on 20.03 at least). In that case we should warn about this issue in our docs (and our release-notes) as this would reappear then after every glibc bump, right?
Can someone explain to me, why exactly this nix-shell -I nixpkgs=`pwd` -p hack works? I'm not sure, I understand entirely, why.

domenkozar commented 4 years ago

Yeah, it happens if you mix 19.09 and 20.03. For example my GHC is coming from 19.09, while ghcide is built with 20.03

neilmayhew commented 4 years ago

So what do you all think about there being no rpath in glibc's libraries? Surely having an rpath is an essential part of the way Nix works? All other libraries seem to have it, and I don't understand why libpthread etc. would be different. Letting something as fundamental as glibc be affected by context seems like a bad idea to me.

Ma27 commented 4 years ago

Not sure yet since I'm wondering if that has been implemented that way for a reason. That's also why I asked if someone can explain why the nix-shell -I hack worked.

neilmayhew commented 4 years ago

It works because nix-shell puts the bin directory of glibc-2.30-bin into PATH and so the ldd that you run is the one from glibc-2.30.

$ git checkout 318f05f
$ which ldd
/run/current-system/sw/bin/ldd
$ readlink -f $(which ldd)
/nix/store/hblpx8x5w88igmwa4ydnsnb65s363lji-glibc-2.27-bin/bin/ldd
$ nix-shell -I nixpkgs=`pwd` -p
$$ which ldd
/nix/store/c2lx3lsqp0y5kwvwkqma8iihc7xjj1m8-glibc-2.30-bin/bin/ldd
$$ echo $PATH | tr : \\n | grep glibc
/nix/store/c2lx3lsqp0y5kwvwkqma8iihc7xjj1m8-glibc-2.30-bin/bin
/nix/store/c2lx3lsqp0y5kwvwkqma8iihc7xjj1m8-glibc-2.30-bin/bin

neilmayhew commented 4 years ago

Also, note that this is purely due to ldd and not to any environment variable. Outside nix-shell:

$ /nix/store/c2lx3lsqp0y5kwvwkqma8iihc7xjj1m8-glibc-2.30-bin/bin/ldd $(nix-build --no-link -I nixpkgs=`pwd` -A glibc)/lib/libpthread.so
  linux-vdso.so.1 (0x00007ffc41184000)
  libc.so.6 => /nix/store/9rabxvqbv0vgjmydiv59wkz768b5fmbc-glibc-2.30/lib/libc.so.6 (0x00007f4c67f73000)
  /nix/store/9rabxvqbv0vgjmydiv59wkz768b5fmbc-glibc-2.30/lib/ld-linux-x86-64.so.2 => /nix/store/9rabxvqbv0vgjmydiv59wkz768b5fmbc-glibc-2.30/lib64/ld-linux-x86-64.so.2 (0x00007f4c68155000)
$ ldd $(nix-build --no-link -I nixpkgs=`pwd` -A glibc)/lib/libpthread.so
  linux-vdso.so.1 (0x00007ffdcbf27000)
  libc.so.6 => /nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/libc.so.6 (0x00007f71789be000)
  /nix/store/9rabxvqbv0vgjmydiv59wkz768b5fmbc-glibc-2.30/lib/ld-linux-x86-64.so.2 => /nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib64/ld-linux-x86-64.so.2 (0x00007f7178b97000)

edolstra commented 4 years ago

Could you do an strace ldd ...? Maybe that will give a hint where the glibc 2.27 dependency comes from.

Note that ldd is implemented by calling ld-linux.so (i.e. it's essentially a wrapper around LD_TRACE_LOADED_OBJECTS=1 /nix/store/.../lib/ld-linux-x86-64.so.2 /path/to/program), so that may be where 2.27 comes from.

nixos-discourse commented 4 years ago

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/go-no-go-meeting-nixos-20-03-markhor/6495/16

neilmayhew commented 4 years ago

With LD_LIBRARY_PATH unset, these seem to be relevant parts of the trace:

7173  execve("/nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib64/ld-linux-x86-64.so.2", ["/nix/store/wx1vk75bpdr65g6xwxbj4"..., "/nix/store/9rabxvqbv0vgjmydiv59w"...], 0x685008 /* 95 vars */) = 0
7173  brk(NULL)                         = 0x55555659b000
7173  openat(AT_FDCWD, "/nix/store/9rabxvqbv0vgjmydiv59wkz768b5fmbc-glibc-2.30/lib/libpthread.so", O_RDONLY|O_CLOEXEC) = 3
7173  read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 l\0\0\0\0\0\0"..., 832) = 832
7173  fstat(3, {st_mode=S_IFREG|0555, st_size=146088, ...}) = 0
7173  mmap(NULL, 131520, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fec1d1e7000
7173  mmap(0x7fec1d1ed000, 61440, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x7fec1d1ed000
7173  mmap(0x7fec1d1fc000, 24576, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15000) = 0x7fec1d1fc000
7173  mmap(0x7fec1d202000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1a000) = 0x7fec1d202000
7173  mmap(0x7fec1d204000, 12736, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fec1d204000
7173  close(3)                          = 0
7173  access("/etc/ld-nix.so.preload", R_OK) = -1 ENOENT (No such file or directory)
7173  openat(AT_FDCWD, "/nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
7173  openat(AT_FDCWD, "/nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/tls/haswell/x86_64/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
7173  stat("/nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/tls/haswell/x86_64", 0x7ffe291a9e40) = -1 ENOENT (No such file or directory)
7173  openat(AT_FDCWD, "/nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/tls/haswell/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
7173  stat("/nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/tls/haswell", 0x7ffe291a9e40) = -1 ENOENT (No such file or directory)
7173  openat(AT_FDCWD, "/nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/tls/x86_64/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
7173  stat("/nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/tls/x86_64", 0x7ffe291a9e40) = -1 ENOENT (No such file or directory)
7173  openat(AT_FDCWD, "/nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/tls/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
7173  stat("/nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/tls", 0x7ffe291a9e40) = -1 ENOENT (No such file or directory)
7173  openat(AT_FDCWD, "/nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/haswell/x86_64/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
7173  stat("/nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/haswell/x86_64", 0x7ffe291a9e40) = -1 ENOENT (No such file or directory)
7173  openat(AT_FDCWD, "/nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/haswell/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
7173  stat("/nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/haswell", 0x7ffe291a9e40) = -1 ENOENT (No such file or directory)
7173  openat(AT_FDCWD, "/nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/x86_64/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
7173  stat("/nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/x86_64", 0x7ffe291a9e40) = -1 ENOENT (No such file or directory)
7173  openat(AT_FDCWD, "/nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
7173  read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260,\2\0\0\0\0\0"..., 832) = 832

Unfortunately argv on the exec is truncated, but it seems like that argv[1] is the location of glibc-2.30/lib/libpthread.so.

So ld-linux-x86-64.so.2 reads libpthread.so and immediately starts looking for its dependencies in glibc-2.27. I assume in the absence of an LD_LIBRARY_PATH and an ld.so.cache it has to fall back on an internally-stored search path, which cannot include glibc-2.30. I don't see how this can possibly work correctly unless libpthread.so has an rpath.

Could somebody please, please tell me why it doesn't have an rpath?

infinisil commented 4 years ago

(Add -s 256 to make it only truncate strings above length 256)

invokesus commented 4 years ago

I have a one-word patch that fixes this(I think).

[132976][04:27:23][inv@nerif:/local/apps/senso/nixpkgs]$ git show HEAD
commit 4c656eb5ca5d34cafdec6429e37a71b203177e32 (HEAD -> ldd)
Author: Souvik Sen <Souvik.Sen@deshaw.com>
Date:   Wed Jan 29 04:14:09 2020 -0500
    glibc.bin: fix ldd output for binaries not compiled with same glibc
    Do not use the explicit dynamic linker this ldd was compiled with.
diff --git a/pkgs/development/libraries/glibc/common.nix b/pkgs/development/libraries/glibc/common.nix
index cdc5dab1be1..f382ba5b20b 100644
--- a/pkgs/development/libraries/glibc/common.nix
+++ b/pkgs/development/libraries/glibc/common.nix
@@ -98,6 +98,9 @@ stdenv.mkDerivation ({
       ./CVE-2018-11236.patch
       # https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=f51c8367685dc888a02f7304c729ed5277904aff
       ./CVE-2018-11237.patch
+
+      # instead of using the linker ldd was compiled with, use the linker the elf specifies
+      ./ldd-use-elf-linker.patch
     ]
     ++ lib.optionals stdenv.isx86_64 [
       ./fix-x64-abi.patch
diff --git a/pkgs/development/libraries/glibc/ldd-use-elf-linker.patch b/pkgs/development/libraries/glibc/ldd-use-elf-linker.patch
new file mode 100644
index 00000000000..747ffa49021
--- /dev/null
+++ b/pkgs/development/libraries/glibc/ldd-use-elf-linker.patch
@@ -0,0 +1,12 @@
+diff -Naur glibc-2.27-orig/elf/ldd.bash.in glibc-2.27/elf/ldd.bash.in
+--- glibc-2.27-orig/elf/ldd.bash.in    2020-01-29 02:17:45.881308184 -0500
++++ glibc-2.27/elf/ldd.bash.in 2020-01-29 02:45:38.057278876 -0500
+@@ -172,7 +172,7 @@
+       }
+       ;;
+     0|2)
+-      try_trace "$RTLD" "$file" || result=1
++      try_trace "$file" || result=1
+       ;;
+     *)
+       echo 'ldd:' ${RTLD} $"exited with unknown exit code" "($ret)" >&2

Wrong output similar to @neilmayhew, before the change.

[132976][04:22:41][inv@nerif:/local/apps/senso/nixpkgs]$ $(nix-build  /nix/var/nix/profiles/per-user/hydra/nixexprs-13000-link/nixpkgs -A glibc.bin)/bin/ldd  /nix/store/ajsrs4bhqclvz2kfv6i4scpv95rcj68z-python-2.7.17/bin/python | tail -1
        /nix/store/519r8kzr2zfyac7qwchlizhafw06wvy7-glibc-2.27/lib/ld-linux-x86-64.so.2 => /nix/store/xv86xix8vr7s1dlhaqdp5j2xdbbwcwgn-glibc-2.27/lib64/ld-linux-x86-64.so.2 (0x00007f4c5f533000)

After the change: Note: <nixpkgs> does not have the patch, so it has a different glibc.

[132976][04:29:43][inv@nerif:/local/apps/senso/nixpkgs]$ $(nix-build -A glibc.bin)/bin/ldd $(nix-build "<nixpkgs>" -A python)/bin/python 
        linux-vdso.so.1 (0x00007ffedc98f000)
        libgcc_s.so.1 => /nix/store/519r8kzr2zfyac7qwchlizhafw06wvy7-glibc-2.27/lib/libgcc_s.so.1 (0x00007f41f5157000)
        libpython2.7.so.1.0 => /nix/store/ajsrs4bhqclvz2kfv6i4scpv95rcj68z-python-2.7.17/lib/libpython2.7.so.1.0 (0x00007f41f4f43000)
        libpthread.so.0 => /nix/store/519r8kzr2zfyac7qwchlizhafw06wvy7-glibc-2.27/lib/libpthread.so.0 (0x00007f41f4f22000)
        libdl.so.2 => /nix/store/519r8kzr2zfyac7qwchlizhafw06wvy7-glibc-2.27/lib/libdl.so.2 (0x00007f41f4f1d000)
        libutil.so.1 => /nix/store/519r8kzr2zfyac7qwchlizhafw06wvy7-glibc-2.27/lib/libutil.so.1 (0x00007f41f4f18000)
        libm.so.6 => /nix/store/519r8kzr2zfyac7qwchlizhafw06wvy7-glibc-2.27/lib/libm.so.6 (0x00007f41f4d82000)
        libc.so.6 => /nix/store/519r8kzr2zfyac7qwchlizhafw06wvy7-glibc-2.27/lib/libc.so.6 (0x00007f41f4bca000)
        /nix/store/519r8kzr2zfyac7qwchlizhafw06wvy7-glibc-2.27/lib/ld-linux-x86-64.so.2 (0x00007f41f536f000)

It also works with impure stuff.

[132976][04:29:38][inv@nerif:/local/apps/senso/nixpkgs]$ $(nix-build -A glibc.bin)/bin/ldd /usr/local/bin/python # RHEL Python
        linux-vdso.so.1 =>  (0x00007ffcf8953000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f0c12287000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f0c12655000)

neilmayhew commented 4 years ago

I think the problem isn't just with ldd, though. This library mismatch cause a number of things to break, as I mentioned in Additional Context.

I would much rather see glibc's .so's get an rpath.

neilmayhew commented 4 years ago

It looks to me like several fixup hooks aren't running in the build of glibc. I don't know yet what's clobbering them.

neilmayhew commented 4 years ago

OK, I see it now. default.nix has NIX_NO_SELF_RPATH = true. I assume a self-rpath is undesirable for libc.so but surely not for the other libs?

neilmayhew commented 4 years ago

Would it work to remove NIX_NO_SELF_RPATH = true and instead use patchelf --remove-rpath libc6.so in postFixup?

vcunat commented 4 years ago

I can't see how adding the RPATH is expected to help things. At the point when you're loading a libpthread, you surely have some version of libc already loaded. I don't think you can that easily have two glibc versions loaded at once anyway (in one process).

vcunat commented 4 years ago

BTW, so far the most common impurity you get (on "desktop use", unless messing with things like $LD_LIBRARY_PATH yourself) comes from OpenGL, i.e. the /run/opengl-driver* path. At least that's my limited perception.

nixos-discourse commented 4 years ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/glibc-problem-with-nixos-rebuild-when-switching-to-nixos-unstable-from-nixos-19-09/6664/1

nixos-discourse commented 4 years ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/xorgserver-upgrade-and-startx/6834/6

nixos-discourse commented 4 years ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/xorgserver-upgrade-and-startx/6834/9

FPtje commented 4 years ago

I'm actually getting the exact same error with Ubuntu 18.04 when using haskell-language-server:

  Unexpected usage error
  /nix/store/7p1v1b6ys9fydg5kdqvr5mpr8svhwf4p-glibc-2.31/lib/libpthread.so: symbol __write_nocancel
  version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference

Except instead of pointing to a libc in the nix store, it points to the one installed by Ubuntu:

$ ldd $(nix-build nixpkgs.nix -A glibc)/lib/libpthread-2.31.so
    linux-vdso.so.1 (0x00007ffca0dec000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3ae3cee000)
    /nix/store/7p1v1b6ys9fydg5kdqvr5mpr8svhwf4p-glibc-2.31/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007f3ae40df000)

Sadly, running the haskell-language-server-wrapper from that shell doesn't fix the error.

This sadly means that it not only happens to people running 19.09, but also people running other Linux OSes.

vcunat commented 4 years ago

I suspect using Ubuntu's ldd on a library won't tell the story right. There's no "interpreter" in *.so files (in the sense of patchelf --print-interpreter), and using a different one (Ubuntu's) won't produce the same result.

stale[bot] commented 3 years ago

I marked this as stale due to inactivity. → More info

nixos-discourse commented 3 years ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/how-do-i-pin-a-specific-version-of-glibc-using-shell-nix/11755/3

ehmry commented 3 years ago

This appears to be a problem with Nim-lang. nix run nimble#about completes on NixOS built frome a recent master but fails on NixOS built from release-20.09.

nixos-discourse commented 3 years ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/fhs-nix-shell-with-older-glibc-2-27-segfaults-on-basic-system-programs/13907/1

stale[bot] commented 2 years ago

I marked this as stale due to inactivity. → More info

jakeisnt commented 1 year ago

Still very much an issue - running into this when trying to pin glibc from nixpkgs.

seanybaggins commented 1 year ago

I am having this issue. I have a shell.nix file where I am trying to pin nixpkgs. When I don't pin nixpkgs, I don't have this issue.

NixOS / nixpkgs

glibc-2.30 libraries link to glibc-2.27 #84043