python3: Enabling optimisations as documented leads to duplicated packages when pulling in aiohttp or filelock

[dnadlinger]

Describe the bug

According to the Nix Python docs, it should be possible to enable optimisations using python3.override. However, this doesn't actually appear to work if some packages are required, among them aiohttp and filelock.

Steps To Reproduce

Try to build the following flake:

{
# Using current master (github:NixOS/nixpkgs/3e644bd62489b516292c816f70bf0052c693b3c7), but
#  inputs.nixpkgs.url = github:NixOS/nixpkgs/nixos-21.11;
# also shows the issue.
  outputs = { self, nixpkgs }:
    let
      pkgs = nixpkgs.legacyPackages.x86_64-linux;
      python = pkgs.python3.override {
          enableOptimizations = true;
          reproducibleBuild = false;
          self = python;
      };
    in {
      defaultPackage.x86_64-linux = python.withPackages (ps: [ps.filelock]);
    };
}

error: builder for '/nix/store/ykm663rbf1d8lm4583kzlwkffy0ifgp1-python3.9-filelock-3.4.2.drv' failed with exit code 1;
       last 10 log lines:
       >   packaging 21.3 (/nix/store/k97wcaz4zyh18dvxcis1xk6ymfsmykqh-python3.9-packaging-21.3/lib/python3.9/site-packages)
       >   packaging 21.3 (/nix/store/4j80m3dgxhdncx5g89gl1h8yia4kf00p-python3.9-packaging-21.3/lib/python3.9/site-packages)
       > Found duplicated packages in closure for dependency 'pyparsing':
       >   pyparsing 3.0.6 (/nix/store/xqbab5gbf3ad8qrg34m88w43y3y6wn43-python3.9-pyparsing-3.0.6/lib/python3.9/site-packages)
       >   pyparsing 3.0.6 (/nix/store/3yyiz59na88bi04h7n36dfqavsg3cs93-python3.9-pyparsing-3.0.6/lib/python3.9/site-packages)
       > Found duplicated packages in closure for dependency 'six':
       >   six 1.16.0 (/nix/store/h4a60ziqsm9n9glv114m3zwhkq5lfi6k-python3.9-six-1.16.0/lib/python3.9/site-packages)
       >   six 1.16.0 (/nix/store/qd8x2fppvsd9fyk06l359fgi8y4wl0hg-python3.9-six-1.16.0/lib/python3.9/site-packages)
       >
       > Package duplicates found in closure, see above. Usually this happens if two packages depend on different version of the same dependency.
       For full logs, run 'nix log /nix/store/ykm663rbf1d8lm4583kzlwkffy0ifgp1-python3.9-filelock-3.4.2.drv'.
error: 1 dependencies of derivation '/nix/store/0lm4xz48a9zn0jmbpjizldi3bc4x3ga5-python3.9-pytest-xdist-2.5.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/0b0jpw2snha5x7yliq5rca67wg38f895-python3.9-hypothesis-6.35.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/vgbf89iinaciyyjixwc9250h2jgw2di5-python3.9-cryptography-36.0.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/rasqxpj48ahhxfmbypidnp6bjrzqv302-python3.9-pytest-asyncio-0.17.2.drv' failed to build
error: 1 dependencies of derivation '/nix/store/vqzfgxgy3ddcs7759qs9gfpbd7zl3knc-python3.9-pytest-mock-3.6.1.drv' failed to build
error: 1 dependencies of derivation '/nix/store/z16vcxi1n9y7h4iy9rr86gag958cim7f-python3.9-aiohttp-3.8.1.drv' failed to build
error: 1 dependencies of derivation '/nix/store/rhdln8jaz5zymbw2ss47aixsng4g6qsj-python3-3.9.10-env.drv' failed to build

Expected behavior

This should build successfully.

Additional context

The nix-store -q --tree output is 2245 lines long, but the two different instances of the libraries are clearly visible: filelock-deps.txt

Not sure I understand what is going on here, but does this potentially involve some cyclic dependencies between Python packages?

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

 - system: `"x86_64-linux"`
 - host os: `Linux 4.15.0-169-generic, Ubuntu, 18.04.5 LTS (Bionic Beaver)`
 - multi-user?: `no`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.6.0`
 - channels(ion): `"nixpkgs-21.05"`
 - nixpkgs: `/home/ion/.nix-defexpr/channels/nixpkgs`

[FRidh]

The issue is the hooks are not being overridden. This results in them providing the default unoptimised versions. Because the other native build input also propagate these packages, but the correct ones, we get collisions.

[FRidh]

Interpreters have an attribute

pythonForBuild = pythonOnBuildForHost.override { inherit packageOverrides; self = pythonForBuild; };

and

pythonOnBuildForHost = pkgsBuildHost.${pythonAttr};

which picks up the build for host interpreter from the spliced package set. In other words, the only way to override correctly is by using an overlay.

Nixpkgs at e1212c4c2812e92d1c7cbed63eb0e417de89a430 and file python.nix:

with import ./. {};

let
  python = python3.override {
    reproducibleBuild = true;
    enableOptimizations = false;
    rebuildBytecode = false;
    enableLTO = false;
    self = python;
  };
in { inherit python; }

[freddy@fr-yoga:~/code/nixpkgs]$ nix-build python.nix -A python                          
/nix/store/i7sapm6pi7323l730kjf54aap7r3awr0-python3-3.9.12

[freddy@fr-yoga:~/code/nixpkgs]$ nix-build python.nix -A python.pythonForBuild
/nix/store/hym1n0ygqp9wcm7pxn4sfrql3fg7xa09-python3-3.9.12

[freddy@fr-yoga:~/code/nixpkgs]$ nix-build -A python3
/nix/store/hym1n0ygqp9wcm7pxn4sfrql3fg7xa09-python3-3.9.12

[freddy@fr-yoga:~/code/nixpkgs]$ nix-build -A python3.pythonForBuild
/nix/store/hym1n0ygqp9wcm7pxn4sfrql3fg7xa09-python3-3.9.12

[FRidh]

Fix in https://github.com/NixOS/nixpkgs/pull/169475.

[dnadlinger]

Thanks!

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/why-is-the-nix-compiled-python-slower/18717/5