New driver with old application fails due to glibc mismatch(?)

Fuuzetsu commented 4 months ago

We have a graphical application that was built with glibc 2.37 at the time. Time has passed, we updated nixpkgs we all use and now people are getting nixGL (nvidia) built against glibc 2.38. When we try to start our application, it fails. The error message from our application is poor and I'm not 100% sure it's a glibc issue but I can replicate something similar with glxgears.

Let's get on old-enough glxgears from nixpkgs: let's use the commit 95dc6a031b8365744241990d0ff51698c1b536d7. This gives me /nix/store/mbsylms2izzlm4w7zqjs0khwvzr1sllp-glxinfo-8.4.0/bin/glxgears.

I then get nixGL built with modern nixpkgs: 037d07f325ad1d811aa3b89e7bf2b0788f645947.

I now try to run modern nixGL against the old glxgears.

[shana@aya:~/programming/engine]$ ldd /nix/store/mbsylms2izzlm4w7zqjs0khwvzr1sllp-glxinfo-8.4.0/bin/glxgears 
    linux-vdso.so.1 (0x00007fffe2f42000)
    libGL.so.1 => /nix/store/yfx0adsr7q8k85w0caswyxrd6bwngbfn-libGL-1.6.0/lib/libGL.so.1 (0x00007fa6da90b000)
    libX11.so.6 => /nix/store/pciixhi1sl4rc4b7wk705qzq8461l14h-libX11-1.8.4/lib/libX11.so.6 (0x00007fa6da7c7000)
    libm.so.6 => /nix/store/0z5kcds7b6qmm373s3b5w9ykvqbgw87i-glibc-2.37-8/lib/libm.so.6 (0x00007fa6da6e7000)
    libc.so.6 => /nix/store/0z5kcds7b6qmm373s3b5w9ykvqbgw87i-glibc-2.37-8/lib/libc.so.6 (0x00007fa6da501000)
    libGLX.so.0 => /nix/store/mw25mnql37bk5j5mbr69s02fp0rc7vcg-libglvnd-1.6.0/lib/libGLX.so.0 (0x00007fa6da4cb000)
    libXext.so.6 => /nix/store/05pz5prswc4i515xihy8yiq26yk8vcg2-libXext-1.3.4/lib/libXext.so.6 (0x00007fa6da4b6000)
    libGLdispatch.so.0 => /nix/store/mw25mnql37bk5j5mbr69s02fp0rc7vcg-libglvnd-1.6.0/lib/libGLdispatch.so.0 (0x00007fa6da3fd000)
    libpthread.so.0 => /nix/store/0z5kcds7b6qmm373s3b5w9ykvqbgw87i-glibc-2.37-8/lib/libpthread.so.0 (0x00007fa6da3f8000)
    libxcb.so.1 => /nix/store/24b0wymzjgl1ylypzj3il3nj2ymf27p4-libxcb-1.14/lib/libxcb.so.1 (0x00007fa6da3cd000)
    /nix/store/0z5kcds7b6qmm373s3b5w9ykvqbgw87i-glibc-2.37-8/lib/ld-linux-x86-64.so.2 => /nix/store/8mc30d49ghc8m5z96yz39srlhg5s9sjj-glibc-2.38-44/lib64/ld-linux-x86-64.so.2 (0x00007fa6da99b000)
    libXau.so.6 => /nix/store/da8b6nbbmck39cnin96phk5mb3kdihps-libXau-1.0.9/lib/libXau.so.6 (0x00007fa6da3c6000)
    libXdmcp.so.6 => /nix/store/rmvg8m2062jd51g5fd9xwhfp0kgwibim-libXdmcp-1.1.3/lib/libXdmcp.so.6 (0x00007fa6da3be000)

[shana@aya:~/programming/engine]$ nixGLNvidia-545.29.06 /nix/store/mbsylms2izzlm4w7zqjs0khwvzr1sllp-glxinfo-8.4.0/bin/glxgears
/nix/store/mbsylms2izzlm4w7zqjs0khwvzr1sllp-glxinfo-8.4.0/bin/glxgears: /nix/store/0z5kcds7b6qmm373s3b5w9ykvqbgw87i-glibc-2.37-8/lib/libc.so.6: version `GLIBC_2.38' not found (required by /nix/store/ncwrra4bx8rl39f4bcs4ykav3y98m2lp-libglvnd-1.7.0/lib/libGLX.so.0)
/nix/store/mbsylms2izzlm4w7zqjs0khwvzr1sllp-glxinfo-8.4.0/bin/glxgears: /nix/store/0z5kcds7b6qmm373s3b5w9ykvqbgw87i-glibc-2.37-8/lib/libc.so.6: version `GLIBC_2.38' not found (required by /nix/store/ncwrra4bx8rl39f4bcs4ykav3y98m2lp-libglvnd-1.7.0/lib/libGLdispatch.so.0)

Welp, that doesn't work. Just to demonstrate that the nixGL install is working, let's try it with modern glxgears from the modern nixpkgs.

[shana@aya:~/programming/engine]$ ldd /nix/store/814qn1iqb9px5i2igwh3sxqdfv2sdzhb-glxinfo-8.4.0/bin/glxgears 
    linux-vdso.so.1 (0x00007ffe36a78000)
    libGL.so.1 => /nix/store/xav6z6xz0n9bb9zvvvivsjyzyzx4lh9q-libGL-1.7.0/lib/libGL.so.1 (0x00007f0b361d2000)
    libX11.so.6 => /nix/store/zdvnf6b6w5sdgxbzg7v6yrlijml8y4yq-libX11-1.8.7/lib/libX11.so.6 (0x00007f0b3608d000)
    libm.so.6 => /nix/store/8mc30d49ghc8m5z96yz39srlhg5s9sjj-glibc-2.38-44/lib/libm.so.6 (0x00007f0b35fab000)
    libc.so.6 => /nix/store/8mc30d49ghc8m5z96yz39srlhg5s9sjj-glibc-2.38-44/lib/libc.so.6 (0x00007f0b35dc2000)
    libGLX.so.0 => /nix/store/ncwrra4bx8rl39f4bcs4ykav3y98m2lp-libglvnd-1.7.0/lib/libGLX.so.0 (0x00007f0b35d8c000)
    libXext.so.6 => /nix/store/7zkcm91ylnlip3xp1kn1xrn7d5zqzq1r-libXext-1.3.6/lib/libXext.so.6 (0x00007f0b35d77000)
    libGLdispatch.so.0 => /nix/store/ncwrra4bx8rl39f4bcs4ykav3y98m2lp-libglvnd-1.7.0/lib/libGLdispatch.so.0 (0x00007f0b35cbe000)
    libpthread.so.0 => /nix/store/8mc30d49ghc8m5z96yz39srlhg5s9sjj-glibc-2.38-44/lib/libpthread.so.0 (0x00007f0b35cb9000)
    libxcb.so.1 => /nix/store/klgd6i59s5dgdv9j8ka78s9ggqp8ijnk-libxcb-1.16/lib/libxcb.so.1 (0x00007f0b35c8e000)
    /nix/store/8mc30d49ghc8m5z96yz39srlhg5s9sjj-glibc-2.38-44/lib/ld-linux-x86-64.so.2 => /nix/store/8mc30d49ghc8m5z96yz39srlhg5s9sjj-glibc-2.38-44/lib64/ld-linux-x86-64.so.2 (0x00007f0b36262000)
    libXau.so.6 => /nix/store/3hqdcdym2v1i2g57xpma73fg5fa4qh3y-libXau-1.0.11/lib/libXau.so.6 (0x00007f0b35c87000)
    libXdmcp.so.6 => /nix/store/srh9dmhzc1zg1cwmndhagxiaswmjnvmp-libXdmcp-1.1.4/lib/libXdmcp.so.6 (0x00007f0b35c7f000)

[shana@aya:~/programming/engine]$ nixGLNvidia-545.29.06 /nix/store/814qn1iqb9px5i2igwh3sxqdfv2sdzhb-glxinfo-8.4.0/bin/glxgears 
Running synchronized to the vertical refresh.  The framerate should be
approximately the same as the monitor refresh rate.

Perfect, so it works.

What I also tried, as I'm on NixOS, is just running both glxgears directly: it works for me! I'm not sure what the magic is. Perhaps I'm still on the old glibc for my system driver and it is forwards compatible but the too-new nixGL doesn't work backwards. To verify this, I installed nixGL against the old nixpkgs. With this, I was able to run both the old and new glxgears. So our current work-around is to pin nixGL against old nixpkgs in our repository and have people install that.

I guess this ticket is a question on what is a good solution to this. Not being able to open old applications kind of sucks. Installing nixGL against old versions also kind of sucks but it's a workaround that could work. Opinions welcome!

Fuuzetsu commented 4 months ago

I ended up finding a version of nixpkgs that

is able to build recent nvidia version with new nixGL repo: if nixpkgs is too old, the new drivers fail to build, presumably the packaging changed mid-way
is able to open applications built recent nixpkgs/glibc
is able to open applications built with nixpkgs from end of 2022

So for now everything works for us. Things will break if drivers change built process and/or nixGL changes in incompatible way with that ancient nixpkgs so the solution is not perfect, but it works for now...

SuperSandro2000 commented 4 months ago

There is nothing that can be done here, don't mix glibc versions.

Fuuzetsu commented 4 months ago

That's what I thought. Shame...

nix-community / nixGL

New driver with old application fails due to glibc mismatch(?) #162