Fixed output derivation produces incorrect nix store path.

benpye commented 1 year ago

Describe the bug

With #5854 being resolved I attempt to convert my nix-fpga-tools to a fixed output derivation again. Unfortunately when making xilinx-ise-unwrapped a fixed output derivation the store path produced, and the store path expected do not match.

Steps To Reproduce

Note: This will require Nix 2.11.0 or later as without the fix for #5854 this would take an unreasonably long amount of time.

Checkout https://github.com/benpye/nix-fpga-tools

Modify xilinx-ise-unwrapped to be fixed output by adding the following to unwrapped.nix:

outputHashMode = "recursive";
outputHashAlgo = "sha256";
outputHash = "b268b731f849a0f93aaaf4b09b198b6376bb41ab8f8897b04f43c4386626a28f";

Run nix shell --verbose .#xilinx-ise and observe it failing to build xilinx-jtag-fw as it attempts to reference a non existent store path.

...
building '/nix/store/h069rn79cnrs77s1209b18s3sh7lcnf5-xilinx-jtag-fw.drv'...
error: builder for '/nix/store/h069rn79cnrs77s1209b18s3sh7lcnf5-xilinx-jtag-fw.drv' failed with exit code 1;
   last 3 log lines:
   > installing
   > cp: missing destination file operand after '/nix/store/qrs9dgcip83fn2q2yzlskhybgizgv7mx-xilinx-jtag-fw/share/'
   > Try 'cp --help' for more information.
   For full logs, run 'nix log /nix/store/h069rn79cnrs77s1209b18s3sh7lcnf5-xilinx-jtag-fw.drv'.
error: 1 dependencies of derivation '/nix/store/0fpkifp8z82891b129b6ig75mp80770c-xilinx-ise-usr-target.drv' failed to build
error: 1 dependencies of derivation '/nix/store/3jmqsnv2g2jsyp7y60a7ga3hdap44a6r-xilinx-ise-fhs.drv' failed to build
error: 1 dependencies of derivation '/nix/store/qw533hqcdciyk48ksp4a7byf50x69yxc-xilinx-ise.drv' failed to build
error: 1 dependencies of derivation '/nix/store/ksr620a8775323xr2lm2zfwbha0h1p7l-xilinx-ise.drv' failed to build

Expected behavior

The build of xilinx-ise succeeds as the only change from the working build is to make xilinx-ise-unwrapped fixed output.

nix-env --version output

[ben@DESKTOP-MOG9R3J:~/git/nix-fpga-tools]$ nix-env --version
nix-env (Nix) 2.11.0

Additional context

After building xilinx-ise-unwrapped derivation as fixed output the store path /nix/store/ajj7pswlap0jxql87a7dfdfss7378vra-xilinx-ise-unwrapped-14.7 is present.

nix show-derivation fails with:

[ben@DESKTOP-MOG9R3J:~/git/nix-fpga-tools]$ nix show-derivation /nix/store/ajj7pswlap0jxql87a7dfdfss7378vra-xilinx-ise-unwrapped-14.7
error: '/nix/store/ajj7pswlap0jxql87a7dfdfss7378vra-xilinx-ise-unwrapped-14.7' does not have a known deriver

nix-store -qd succeeds showing:

[ben@DESKTOP-MOG9R3J:~/git/nix-fpga-tools]$ nix-store -qd /nix/store/ajj7pswlap0jxql87a7dfdfss7378vra-xilinx-ise-unwrapp
ed-14.7
/nix/store/1f1mlg063mwhrn95hi5d5rh15ykafdmz-xilinx-ise-unwrapped-14.7.drv

Getting the derivation for the .drv gives a different expected out path, and this is the path that dependent derivations attempt to use:

[ben@DESKTOP-MOG9R3J:~/git/nix-fpga-tools]$ nix show-derivation /nix/store/1f1mlg063mwhrn95hi5d5rh15ykafdmz-xilinx-ise-unwrapped-14.7.drv
{
"/nix/store/1f1mlg063mwhrn95hi5d5rh15ykafdmz-xilinx-ise-unwrapped-14.7.drv": {
"outputs": {
  "out": {
    "path": "/nix/store/g04ycpg8j1lchh9w4804ych8v95288yl-xilinx-ise-unwrapped-14.7",
    "hashAlgo": "r:sha256",
    "hash": "b268b731f849a0f93aaaf4b09b198b6376bb41ab8f8897b04f43c4386626a28f"
  }
},
"inputSrcs": [
  "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"
],
"inputDrvs": {
  "/nix/store/25v14drl573yalys9kdc4c1m8kz5fmvd-bash-5.1-p16.drv": [
    "out"
  ],
  "/nix/store/4av1cpcrlcnh3cmhnl9y9fhfripg259w-7zz-21.07.drv": [
    "out"
  ],
  "/nix/store/4zg4z7730kn3z0wsc0x3ax04wsbqr9c3-Xilinx_ISE_14.7_Win10_14.7_VM_0213_1.zip.drv": [
    "out"
  ],
  "/nix/store/br9cklh50wmzvkdrymd4lkw2b2h0cybg-stdenv-linux.drv": [
    "out"
  ],
  "/nix/store/mzydkdp3zlgli5h6ir7v2rznrfam1h6x-unzip-6.0.drv": [
    "out"
  ]
},
"system": "x86_64-linux",
"builder": "/nix/store/9zm6br2ri10a0b71dll2wrim5bnhg6b6-bash-5.1-p16/bin/bash",
"args": [
  "-e",
  "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"
],
"env": {
  "buildCommand": "unzip $src\ntar xvf ova/14.7_VM.ova\nrm ova/14.7_VM.ova\n7zz e 14.7_VM-disk001.vmdk 0.img\nrm 14.7_VM-disk001.vmdk\n7zz x -snld -o$out 0.img opt/Xilinx home/ise\n",
  "buildInputs": "",
  "builder": "/nix/store/9zm6br2ri10a0b71dll2wrim5bnhg6b6-bash-5.1-p16/bin/bash",
  "configureFlags": "",
  "depsBuildBuild": "",
  "depsBuildBuildPropagated": "",
  "depsBuildTarget": "",
  "depsBuildTargetPropagated": "",
  "depsHostHost": "",
  "depsHostHostPropagated": "",
  "depsTargetTarget": "",
  "depsTargetTargetPropagated": "",
  "doCheck": "",
  "doInstallCheck": "",
  "name": "xilinx-ise-unwrapped-14.7",
  "nativeBuildInputs": "/nix/store/g96i1pjsfhfbp7553851zvvl18fr3kgv-unzip-6.0 /nix/store/jz7f71z533r1w353aa6bx2ndm84snh42-7zz-21.07",
  "out": "/nix/store/g04ycpg8j1lchh9w4804ych8v95288yl-xilinx-ise-unwrapped-14.7",
  "outputHash": "b268b731f849a0f93aaaf4b09b198b6376bb41ab8f8897b04f43c4386626a28f",
  "outputHashAlgo": "sha256",
  "outputHashMode": "recursive",
  "outputs": "out",
  "patches": "",
  "pname": "xilinx-ise-unwrapped",
  "propagatedBuildInputs": "",
  "propagatedNativeBuildInputs": "",
  "src": "/nix/store/8x0b9fsr4jh9ii3v0zl3h42qs8288caq-Xilinx_ISE_14.7_Win10_14.7_VM_0213_1.zip",
  "stdenv": "/nix/store/3b0i6992k56lzqpndnhim5pbr3wnfnjx-stdenv-linux",
  "strictDeps": "",
  "system": "x86_64-linux",
  "version": "14.7"
}
}
}

ncfavier commented 1 year ago

I just hit this today.

# test.nix
with import <nixpkgs> {};
fetchFromGitHub {
  owner = "NixOS";
  repo = "nix";
  rev = "master";
  postFetch = ''
    echo ${hello} > $out/foo
  '';
  hash = "sha256-cYnCDbIID6qZzpzu9ax6KfGiQ9JaQkc6GnXU7qx6B7k=";
}

$ nix build -f test.nix --print-out-paths
/nix/store/p30525s24kk8gcdgjpgjqra78fam66sx-source
$ nix eval -f test.nix outPath
"/nix/store/3k03r6dp1hpw1k8lw786ibdyc7bv07s1-source"

Same with the nix-build/nix-instantiate CLI.

This only happens if a derivation is added to the output (here hello).

If I use a path instead (echo ${/some/file} > $out/foo), I get a never-ending sequence of "hash mismatch" errors.

ncfavier commented 1 year ago

Minimal reproducer:

with import <nixpkgs> {};
let
  mkDerivation = args:
    derivation ({
      system = "x86_64-linux";
      builder = "${bash}/bin/bash";
      PATH = "${coreutils}/bin";
      args = ["-e" "-c" "eval \"$buildCommand\""];
    } // args);

  dep = mkDerivation {
    name = "dep";
    buildCommand = "echo foo > $out";
  };
in
mkDerivation {
  name = "test";
  buildCommand = ''
    mkdir -p $out
    echo ${dep} > $out/dep
  '';
  outputHashMode = "recursive";
  outputHashAlgo = "sha256";
  outputHash = "sha256-rGi3RZBPE6sy5Np2EFQKKPmiH9foV25h+f0RUwezLng=";
}

Using builtins.trace to print the outPath agrees with nix eval, while the actual build output is under the path printed by nix build.

Weirdly enough I can't reproduce this in a Nix test, the two commands agree.

@thufschmitt any idea what's going on?

tomberek commented 1 year ago

@ncfavier That output hash seems wrong. It may be colliding with a store path you already have, and thus the build short-circuits, but the eval just tells you what the outPath should be. Run the commands with an empty store, --store, or otherwise invalidate the hash to check.

ncfavier commented 1 year ago

That's the hash Nix gives me after I try to build with an empty hash. Obviously it depends on stuff in nixpkgs so it might have changed now. Building in an empty store doesn't change anything.

ncfavier commented 1 year ago

Seems like this is a duplicate of https://github.com/NixOS/nix/issues/6660

edolstra commented 3 months ago

Fixed in current Nix, e.g. the reproducer above gives

error: illegal path references in fixed-output derivation '/nix/store/h46iklv790mwzrr4kivhrdbf8glql0l6-test.drv'

nixos-discourse commented 3 months ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-06-12-nix-team-meeting-minutes-152/46958/1

NixOS / nix

Fixed output derivation produces incorrect nix store path. #7150