executable fails to launch during installPhase on Linux - works later from nix-shell

Describe the bug

I have created a custom derivation that includes a binary build of python that we build locally. It works when run from the nix shell, but fails when launching from the installPhase for my derivation - only on Linux.

EXPECTED_RESULT: I am able to launch the binary from the installPhase so that I can add more python modules to the installation ACTUAL_RESULT: without strace I get the error: "required file not found" from the installPhase shell. With strace, execve(...) cannot find the python binary - either way I cannot run it.

Operating system:

bash-4.2$ uname -a Linux sc-dbc9900 3.10.0-1160.95.1.el7.x86_64 #1 SMP Mon Jul 24 13:59:37 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux bash-4.2$ cat /etc/os-release NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7"

Steps To Reproduce

Steps to reproduce the behavior:

Try to launch a binary from installPhase before nix-shell is completely launched - but after all of the patchelf ... commands are run

EXPECTED_RESULT: Binary should launch ACTUAL_RESULT: Required file not found.

Expected behavior

Fully patched binaries should be launchable from the install phase

Screenshots

this derivation will be built:
  /nix/store/jb8641gyfagdq547iqx2d8c6bib2lyg9-python-3.11.drv
building '/nix/store/jb8641gyfagdq547iqx2d8c6bib2lyg9-python-3.11.drv'...
unpacking sources
unpacking source archive /nix/store/j9da8xl2h6k2b3d7b25bfgj816fiik10-source
source root is source
patching sources
configuring
no configure script, doing nothing
building
no Makefile or custom buildPhase, doing nothing
installing
patchelf --replace-needed libpython3.11.so.1.0 /nix/store/yi5j7cf5xx031k2cbar6mkk41hss9x7y-python-3.11/python/lib/libpython3.11.so.1.0 /nix/store/yi5j7cf5xx031k2cbar6mkk41hss9x7y-python-3.11/python/bin/python3.11
patchelf --replace-needed libz.so.1 /nix/store/69jpyha5zbll6ppqzhbihhp51lac1hrp-zlib-1.2.13/lib/libz.so.1 /nix/store/yi5j7cf5xx031k2cbar6mkk41hss9x7y-python-3.11/python/lib/libpython3.11.so.1.0
patchelf --replace-needed libssl.so.3 /nix/store/xhsm7fs0fik5vc2bv7j4xa1ibdx6jbaj-openssl-3.0/openssl/lib/libssl.so.3 --replace-needed libcrypto.so.3 /nix/store/xhsm7fs0fik5vc2bv7j4xa1ibdx6jbaj-openssl-3.0/openssl/lib/libcrypto.so.3 /nix/store/yi5j7cf5xx031k2cbar6mkk41hss9x7y-python-3.11/python/lib/python3.11/lib-dynload/_ssl.cpython-311-x86_64-linux-gnu.so
patchelf --replace-needed libcrypto.so.3 /nix/store/xhsm7fs0fik5vc2bv7j4xa1ibdx6jbaj-openssl-3.0/openssl/lib/libcrypto.so.3 /nix/store/yi5j7cf5xx031k2cbar6mkk41hss9x7y-python-3.11/python/lib/python3.11/lib-dynload/_hashlib.cpython-311-x86_64-linux-gnu.so
ldd /nix/store/yi5j7cf5xx031k2cbar6mkk41hss9x7y-python-3.11/python/bin/python3.11
        linux-vdso.so.1 (0x00007ffff7fc8000)
        /nix/store/yi5j7cf5xx031k2cbar6mkk41hss9x7y-python-3.11/python/lib/libpython3.11.so.1.0 (0x00007ffff79f7000)
        libc.so.6 => /nix/store/46m4xx889wlhsdj72j38fnlyyvvvvbyb-glibc-2.37-8/lib/libc.so.6 (0x00007ffff7811000)
        /nix/store/69jpyha5zbll6ppqzhbihhp51lac1hrp-zlib-1.2.13/lib/libz.so.1 (0x00007ffff77f2000)
        libpthread.so.0 => /nix/store/46m4xx889wlhsdj72j38fnlyyvvvvbyb-glibc-2.37-8/lib/libpthread.so.0 (0x00007ffff77ed000)
        libdl.so.2 => /nix/store/46m4xx889wlhsdj72j38fnlyyvvvvbyb-glibc-2.37-8/lib/libdl.so.2 (0x00007ffff77e8000)
        libutil.so.1 => /nix/store/46m4xx889wlhsdj72j38fnlyyvvvvbyb-glibc-2.37-8/lib/libutil.so.1 (0x00007ffff77e1000)
        libm.so.6 => /nix/store/46m4xx889wlhsdj72j38fnlyyvvvvbyb-glibc-2.37-8/lib/libm.so.6 (0x00007ffff7701000)
        /lib64/ld-linux-x86-64.so.2 => /nix/store/46m4xx889wlhsdj72j38fnlyyvvvvbyb-glibc-2.37-8/lib64/ld-linux-x86-64.so.2 (0x00007ffff7fca000)
patchelf --print-interpreter --print-soname --print-rpath --print-needed /nix/store/yi5j7cf5xx031k2cbar6mkk41hss9x7y-python-3.11/python/bin/python3.11
/lib64/ld-linux-x86-64.so.2
$ORIGIN/../lib
/nix/store/yi5j7cf5xx031k2cbar6mkk41hss9x7y-python-3.11/python/lib/libpython3.11.so.1.0
libc.so.6
execve("/nix/store/yi5j7cf5xx031k2cbar6mkk41hss9x7y-python-3.11/python/bin/python3.11", ["/nix/store/yi5j7cf5xx031k2cbar6m"...], 0x7fffffffc590 /* 79 vars */) = -1 ENOENT (No such file or directory)
/nix/store/fqj5bn47jb3dhijjgdrwhbzsbjf7831v-strace-6.4/bin/strace: exec: No such file or directory
+++ exited with 1 +++

Additional context

It seems that somehow the executable loader for the operating system cannot see the /nix/store somehow during the installPhase.

Here is the full source for my derivation:

{ pkgs, cayman_openssl3, localModules }:

let
  addLocalModules = import ./add_local_modules.nix;
  build = "sb-67336917";
  pythonSettings = if pkgs.stdenv.system == "x86_64-darwin" then
    {
      arch = "apple_mac64+nix";
      sha = "sha256:13kf1k1ppp923s0ash2zkzws62rvx6qsd279krbzah2i2b2iiyx3";
      extra_commands = ''
      # update the dylib location
      echo install_name_tool  \
        -change  /Volumes/build/mts/release/${build}/cayman_python/build/release/apple_mac64+nix/python/install/lib/libpython3.11.dylib    $out/python/lib/libpython3.11.dylib \
        $out/python/bin/python3.11
      install_name_tool  \
        -change  /Volumes/build/mts/release/${build}/cayman_python/build/release/apple_mac64+nix/python/install/lib/libpython3.11.dylib   $out/python/lib/libpython3.11.dylib \
        $out/python/bin/python3.11
      echo install_name_tool \
        -id $out/python/lib/libpython3.11.dylib $out/python/lib/libpython3.11.dylib
      install_name_tool \
        -id $out/python/lib/libpython3.11.dylib $out/python/lib/libpython3.11.dylib
      echo install_name_tool \
        -change @GOBUILD_CAYMAN_OPENSSL_ROOT@/apple_macos-universal/lib/libssl.3.dylib ${cayman_openssl3}/openssl/lib/libssl.3.dylib \
        -change @GOBUILD_CAYMAN_OPENSSL_ROOT@/apple_macos-universal/lib/libcrypto.3.dylib ${cayman_openssl3}/openssl/lib/libcrypto.3.dylib \
        $out/python/lib/python3.11/lib-dynload/_ssl.cpython-311-darwin.so
      install_name_tool \
        -change @GOBUILD_CAYMAN_OPENSSL_ROOT@/apple_macos-universal/lib/libssl.3.dylib ${cayman_openssl3}/openssl/lib/libssl.3.dylib \
        -change @GOBUILD_CAYMAN_OPENSSL_ROOT@/apple_macos-universal/lib/libcrypto.3.dylib ${cayman_openssl3}/openssl/lib/libcrypto.3.dylib \
        $out/python/lib/python3.11/lib-dynload/_ssl.cpython-311-darwin.so
      echo install_name_tool \
        -change @GOBUILD_CAYMAN_OPENSSL_ROOT@/apple_macos-universal/lib/libcrypto.3.dylib ${cayman_openssl3}/openssl/lib/libcrypto.3.dylib \
        $out/python/lib/python3.11/lib-dynload/_hashlib.cpython-311-darwin.so
      install_name_tool \
        -change @GOBUILD_CAYMAN_OPENSSL_ROOT@/apple_macos-universal/lib/libcrypto.3.dylib ${cayman_openssl3}/openssl/lib/libcrypto.3.dylib \
        $out/python/lib/python3.11/lib-dynload/_hashlib.cpython-311-darwin.so
      '';
    } else {
      arch = "lin64+glibc217+gcc12+nix";
      sha = "sha256:1xfjcqiz71na0rdfqz9l2n2bbiqkdavvxfcpnrm2m6lrgqq9k8hf";
      extra_commands = ''
        # Remove broken symlink
        rm $out/python/lib/libpython3.11.so
        # Re-create symlink
        ln -s $out/python/lib/libpython3.11.so.1.0 $out/python/lib/libpython3.11.so
        # patch executable .so locations
        echo patchelf --replace-needed libpython3.11.so.1.0 $out/python/lib/libpython3.11.so.1.0 $out/python/bin/python3.11
        patchelf --replace-needed libpython3.11.so.1.0 $out/python/lib/libpython3.11.so.1.0 $out/python/bin/python3.11
        echo patchelf --replace-needed libz.so.1 ${pkgs.zlib}/lib/libz.so.1 $out/python/lib/libpython3.11.so.1.0
        patchelf --replace-needed libz.so.1 ${pkgs.zlib}/lib/libz.so.1 $out/python/lib/libpython3.11.so.1.0
        echo patchelf \
          --replace-needed libssl.so.3 ${cayman_openssl3}/openssl/lib/libssl.so.3  \
          --replace-needed libcrypto.so.3 ${cayman_openssl3}/openssl/lib/libcrypto.so.3 \
          $out/python/lib/python3.11/lib-dynload/_ssl.cpython-311-x86_64-linux-gnu.so
        patchelf \
          --replace-needed libssl.so.3 ${cayman_openssl3}/openssl/lib/libssl.so.3  \
          --replace-needed libcrypto.so.3 ${cayman_openssl3}/openssl/lib/libcrypto.so.3 \
          $out/python/lib/python3.11/lib-dynload/_ssl.cpython-311-x86_64-linux-gnu.so
        echo patchelf \
          --replace-needed libcrypto.so.3 ${cayman_openssl3}/openssl/lib/libcrypto.so.3 \
          $out/python/lib/python3.11/lib-dynload/_hashlib.cpython-311-x86_64-linux-gnu.so
        patchelf \
          --replace-needed libcrypto.so.3 ${cayman_openssl3}/openssl/lib/libcrypto.so.3 \
          $out/python/lib/python3.11/lib-dynload/_hashlib.cpython-311-x86_64-linux-gnu.so
      '';
    };
    addLocalModulesCommands = addLocalModules { modules=localModules; };
in
pkgs.stdenv.mkDerivation rec {
  pname = "python";
  version = "3.11";

  src = fetchTarball {
    url="http://build-squid.eng.vmware.com/build/mts/release/${build}/publish/python-${pythonSettings.arch}.tgz";
    sha256= pythonSettings.sha;
  };

  buildInputs = [
    cayman_openssl3
    # TODO (MGIT-423): Update to use cayman_zlib instead of nixos.zlib
    pkgs.zlib
    pkgs.strace
  ];

  installPhase = ''
    mkdir -p $out/bin
    mkdir -p $out/python
    # Remove broken links
    rm bin/2to3
    rm bin/idle3
    rm bin/pydoc3
    rm -f bin/python
    rm bin/python3
    rm bin/python3-config
    rm -f bin/python3-intel64
    cp -rf * $out/python
    # Setup basic links into the $out/bin directory
    ln -s $out/python/bin/python3.11 $out/bin/python3.11
    ln -s $out/python/bin/2to3-3.11 $out/bin/2to3-3.11
    ln -s $out/python/bin/pip3.11 $out/bin/pip3.11
    ln -s $out/python/bin/pydoc3.11 $out/bin/pydoc3.11
    ln -s $out/python/bin/python3.11-config $out/bin/python3.11-config
    ln -s $out/python/bin/idle3.11 $out/bin/idle3.11
    # Restore python3 link to the python directory
    # ln -s $out/python/bin/python3.11 $out/python/bin/python3
    # Extra spellings for python
    ln -s $out/bin/python3.11 $out/bin/python3
    ln -s $out/bin/python3 $out/bin/python
    # Extra spellings for 2to3
    ln -s $out/bin/2to3-3.11 $out/bin/2to3
    # Extra spellings for pip
    ln -s $out/bin/pip3.11 $out/bin/pip3
    ln -s $out/bin/pip3 $out/bin/pip
    # Extra spellings for pydoc
    ln -s $out/bin/pydoc3.11 $out/bin/pydoc3
    ln -s $out/bin/pydoc3 $out/bin/pydoc
    # Extra spellings for python-config
    ln -s $out/bin/python3.11-config $out/bin/python3-config
    ln -s $out/bin/python3-config $out/bin/python-config
    # Extra spellings for idle
    ln -s $out/bin/idle3.11 $out/bin/idle3
    ln -s $out/bin/idle3 $out/bin/idle
    # Fix #! lines
    sed -i "s|#\!.*$|#\!env python3.11|" $out/python/bin/2to3-3.11
    sed -i "s|#\!.*$|#\!env python3.11|" $out/python/bin/idle3.11
    sed -i "s|#\!.*$|#\!env python3.11|" $out/python/bin/pip3
    sed -i "s|#\!.*$|#\!env python3.11|" $out/python/bin/pip3.11
    sed -i "s|#\!.*$|#\!env python3.11|" $out/python/bin/pydoc3.11
    sed -i "s|#\!.*$|#\!env python3.11|" $out/python/bin/python3.11-config
    ${pythonSettings.extra_commands}
    echo  ldd $out/python/bin/python3.11
    ldd $out/python/bin/python3.11
    echo patchelf --print-interpreter --print-soname --print-rpath --print-needed  $out/python/bin/python3.11
    patchelf --print-interpreter --print-soname --print-rpath --print-needed  $out/python/bin/python3.11
    ${pkgs.strace}/bin/strace $out/python/bin/python3.11
  '';

  #postInstall = ''
  #  ldd $out/python/bin/python3.11
  #  $out/python/bin/python3.11 -m pip install --no-warn-script-location --upgrade pip
  #  ${addLocalModulesCommands}
  #'';

}

Unfortunately, you cannot run this derivation because it uses an internal build system to source the outputs.

Notify maintainers

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
bash-4.2$ nix-shell -p nix-info --run "nix-info -m"
this path will be fetched (0.00 MiB download, 0.00 MiB unpacked):
  /nix/store/k4996pr5b2nfmxyvzbm8r5wysz9b10ii-nix-info
copying path '/nix/store/k4996pr5b2nfmxyvzbm8r5wysz9b10ii-nix-info' from 'https://cache.nixos.org'...
 - system: `"x86_64-linux"`
 - host os: `Linux 3.10.0-1160.95.1.el7.x86_64, CentOS Linux, 7 (Core), nobuild`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.17.0`
 - channels(root): `"nixpkgs"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixpkgs`

NOTE: I am using nixos-23.05 as the source nix distribution pinned at: SHA: 475d5ae2c4cb87b904545bdb547af05681198fcc

NixOS / nixpkgs