An unexpected error (like a segfault) collecting package requirements leads to a bogus requirement store object

[exarkun]

I did a build which ended like

building '/nix/store/8b2c2p6vvm3n1hzz89kpdm2nxx5h3icd-package-requirements.drv'...                                                                                                
unpacking sources                                                                                                                                                                 
unpacking source archive /nix/store/jqnygi353wwb9cj5spg3f3klq853qzsm-source                                                                                                       
source root is source                                                                                                                                                             
installing                                                                                                                                                                        
extracting dependencies                                                                                                                                                           
/nix/store/7r9phvkav218az7sl7q3qkyka7i0p51b-stdenv-linux/setup: line 1381:    18 Segmentation fault      (core dumped) out_file=$out/python.json /nix/store/8fp51f2f2xarssv6l835lc
ji73vfjvr0-python3-3.7.13-env-patched/bin/python -c "import sys, setuptools, tokenize, os; sys.argv[0] = 'setup.py'; __file__='setup.py';                                         
f=getattr(tokenize, 'open', open)(__file__);                                                                                                                                      
code=f.read().replace('\r\n', '\n');                                                                                                                                              
f.close();                                                                                                                                                                        
exec(compile(code, __file__, 'exec'))                                                                                                                                             
" install &> $out/python.log                                                                                                                                                      
error: Automatic extraction of 'pname' from python package source /nix/store/jqnygi353wwb9cj5spg3f3klq853qzsm-source failed.                                                      
       Please manually specify 'pname'                                                                                                                                            
(use '--show-trace' to show detailed location information)                               

I tried the build again and it failed with only this output

error: Automatic extraction of 'pname' from python package source /nix/store/jqnygi353wwb9cj5spg3f3klq853qzsm-source failed.
       Please manually specify 'pname'
(use '--show-trace' to show detailed location information)

I checked on the state of the package-requirements derivation from the first build output:

nix-build /nix/store/8b2c2p6vvm3n1hzz89kpdm2nxx5h3icd-package-requirements.drv
/nix/store/1p2haa71z817ylr2z66vfc3q1rbbfczh-package-requirements

Ah. It finished building and is now in the store. What's in it?

$ ls -lR /nix/store/1p2haa71z817ylr2z66vfc3q1rbbfczh-package-requirements
/nix/store/1p2haa71z817ylr2z66vfc3q1rbbfczh-package-requirements:
total 0
-r--r--r-- 1 root root 0 Dec 31  1969 python.log

Oh, nothing.

It seems like the segfault that occurred during building package-requirements.drv should have failed the build?

[bjornfor]

I came across this myself and bisected a bit.

The first bad nixpkgs commit is https://github.com/nixos/nixpkgs/commit/5c09870c0244bbcf47a84f379e05bf10c7aa3f0d ("python3Packages.setuptools: 57.2.0 -> 60.8.2 ").

Then I bisected setuptools and found https://github.com/pypa/setuptools/commit/b6fcbbd00cb6d5607c9272dec452a50457bdb292 to be the first bad commit.

Looking at the setuptools commit I thought I could fix it quickly by exporting SETUPTOOLS_USE_DISTUTILS=stdlib, to restore the old behaviour, but that didn't seem to work. (Or I messed up setting that variable inside the derivation.)

[exarkun]

Thank you for investigating and reporting your results @bjornfor !

[bjornfor]

Here's a repro:

let
  nixpkgs = import (builtins.fetchGit {
    url = "https://github.com/nixos/nixpkgs";
      ref = "refs/heads/nixos-22.05";
      rev = "9ff91ce2e4c5d70551d4c8fd8830931c6c6b26b8";
  }) { config = {}; overlays = []; };

  mach-nix = import (builtins.fetchGit {
    url = "https://github.com/DavHau/mach-nix";
    ref = "refs/heads/master";
    rev = "7e14360bde07dcae32e5e24f366c83272f52923f"; # aka. 3.5.0
  }) { pkgs = nixpkgs; };
in
  # Build an arbitrary package, but one that requires introspecting the deps,
  # and using recent nixpkgs.
  mach-nix.buildPythonPackage {
    src = builtins.fetchGit {
      url = "https://github.com/discogs/python-cas-client.git";
      ref = "refs/heads/master";
      rev = "f1efa2f49a22d43135014cb1b8d9dd3875304318";
    };
  }

$ nix-build ./repro.nix
[...]
extracting dependencies
/nix/store/7r9phvkav218az7sl7q3qkyka7i0p51b-stdenv-linux/setup: line 1381:    20 Segmentation fault      (core dumped) out_file=$out/python.json /nix/store/siyq47lvs205nisg8zgnkf2i4dyjn0gm-python3-3.9.13-env-patched/bin/python -c "import sys, setuptools, tokenize, os; sys.argv[0] = 'setup.py'; __file__='setup.py';
f=getattr(tokenize, 'open', open)(__file__);
code=f.read().replace('\r\n', '\n');
f.close();
exec(compile(code, __file__, 'exec'))
" install &> $out/python.log
copying 1 paths...
copying path '/nix/store/606xjwiyylmw26fq9hdaysqyazqw8jpa-package-requirements' from 'ssh://nix-remote-build@bforsman.name'...
error: Automatic extraction of 'pname' from python package source /nix/store/ldgqv3yws25xaa1akfmya1sqf7vxsgq0-source failed.
Please manually specify 'pname' 
(use '--show-trace' to show detailed location information)

[TyberiusPrime]

I've debuged it down to a) the setuptools change requiring SETUPTOOLS_USE_DISTUTILS=stdlib, and then mach-nix trying to patch the python-wrapper-script by sed-ing, which of course breaks a binary wrapper.