"error: path '/nix/store/...' is not valid" when saving a reference to a store path in $out in a fixed output derivation

fgaz commented 2 years ago

Describe the bug

I get the (not very clear) error in the title for no apparent reason

Steps To Reproduce

nix-build this file

{ pkgs ? import <nixpkgs> {} }:

let dep = pkgs.stdenv.mkDerivation {
  name = "dep";

  src = pkgs.runCommand "src" {} "mkdir $out && touch $out/file;";

  installPhase = "echo $src > $out";

  outputHashMode = "recursive";
  outputHash = "sha256-yQ372mYt9JuOW1i0nprYCoMQDUjZsE561vlnHQFyuew=";
};

in pkgs.runCommand "cmd" {} "echo ${dep} && touch $out"

result:

these 2 derivations will be built:
  /nix/store/lg8r8cziaw3q3j0mvwl6am2ibba1d09m-dep.drv
  /nix/store/k6ha4y9fcvzhfpj331hzbkb432r0a5xr-cmd.drv
building '/nix/store/lg8r8cziaw3q3j0mvwl6am2ibba1d09m-dep.drv'...
unpacking sources
unpacking source archive /nix/store/53czgxl1mmaspkl4cxc1m26jmwsbisl7-src
source root is src
patching sources
configuring
no configure script, doing nothing
building
no Makefile, doing nothing
installing
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/1z2mjgxil1skrv31j26v94kzpxmz11p6-dep
strip is /nix/store/a4mmjm3bblxwp8h53bcfx3dly80ib0ba-binutils-2.35.1/bin/strip
patching script interpreter paths in /nix/store/1z2mjgxil1skrv31j26v94kzpxmz11p6-dep
checking for references to /build/ in /nix/store/1z2mjgxil1skrv31j26v94kzpxmz11p6-dep...
error: path '/nix/store/1z2mjgxil1skrv31j26v94kzpxmz11p6-dep' is not valid

edit: even shorter repro, demonstrating that the issue doesn't have anything to do with src:

{ pkgs ? import <nixpkgs> {} }:
let
  src = pkgs.runCommand "src" {} "mkdir $out && touch $out/file;";
  dep = pkgs.runCommand "dep" {
    outputHashMode = "recursive";
    outputHash = "sha256-yQ372mYt9JuOW1i0nprYCoMQDUjZsE561vlnHQFyuew=";
  } "echo ${src} > $out";
in pkgs.runCommand "cmd" {} "echo ${dep} && touch $out"

Expected behavior

I expected the command to succeed

nix-env --version output

nix-env (Nix) 2.4pre20210601_5985b8b

Additional context

vvs- commented 2 years ago

I'm not an expert but isn't this counts as IFD which is a hot topic in Nix 2.4?

fgaz commented 2 years ago

At this point I'm not sure either, but nix-instantiate does not build anything for me so it looks like it isn't.

As far as I can see my code does not use the contents of a derivation to build another one, it only uses its path. Is it because the path of a fod depends on its contents? Isn't the hash there exactly to avoid that dependency (and for reproducibility of course)?

vvs- commented 2 years ago

You should be able to test this hypothesis by passing --allow-import-from-derivation to Nix. Also, it would be beneficial to use released Nix 2.4 instead of old pre-released version.

bergkvist commented 2 years ago

I was just about to post the same issue! (nix-env (Nix) 2.3.10). I'm on macOS Big Sur.

{ pkgs ? import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/afdb5675a180f347bfa8ae909d4e419fb8b151bd.tar.gz") {}
}: pkgs.stdenv.mkDerivation {
  name = "test";
  outputHashMode = "recursive";
  outputHashAlgo = "sha256";
  outputHash = pkgs.lib.fakeSha256;
  phases = [ "installPhase" ];
  installPhase = ''
    mkdir -p "$out"
    cd "$out"
    echo "$out" > x
  '';
}

these derivations will be built:
  /nix/store/0wb862jxlhqqkcn8z83f00zjvv7ya94i-test.drv
building '/nix/store/0wb862jxlhqqkcn8z83f00zjvv7ya94i-test.drv'...
installing
error: path '/nix/store/2rlnd8mvb8ccijspylciscq1q0ka8zhf-test' is not valid

If I put something else into the output file:

{ pkgs ? import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/afdb5675a180f347bfa8ae909d4e419fb8b151bd.tar.gz") {}
}: pkgs.stdenv.mkDerivation {
  name = "test";
  outputHashMode = "recursive";
  outputHashAlgo = "sha256";
  outputHash = pkgs.lib.fakeSha256;
  phases = [ "installPhase" ];
  installPhase = ''
    mkdir -p "$out"
    cd "$out"
    echo "does-not-contain-output-path" > x
  '';
}

these derivations will be built:
  /nix/store/ah3zvkdb735ig7a8068hqqcsp08lzzjy-test.drv
building '/nix/store/ah3zvkdb735ig7a8068hqqcsp08lzzjy-test.drv'...
installing
hash mismatch in fixed-output derivation '/nix/store/5fyk5lssqs4wcy0lk0l9ca95lz0p0ivb-test':
  wanted: sha256:0000000000000000000000000000000000000000000000000000
  got:    sha256:180wmzj5dabx60nr8dh964py4ykvbg3j7xpdhgbfspk7ls06i34d
error: build of '/nix/store/ah3zvkdb735ig7a8068hqqcsp08lzzjy-test.drv' failed

Which is what I want to see. Here I actually get a hash back. This issue is only present when I use fixed-output derivations (where I specify an output hash).

Context

I'm trying to use pip to install Python libraries - but pip writes the install path into .pyc-files in __pycache__-folders. This causes the derivation to fail building with "path XXXXX is not valid".

bergkvist commented 2 years ago

On nix-env (Nix) 2.4pre20210601_5985b8b (Manjaro Linux), I don't actually get any error during nix-build:

{ pkgs ? import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/afdb5675a180f347bfa8ae909d4e419fb8b151bd.tar.gz") {}
}: pkgs.stdenv.mkDerivation {
  name = "test";
  outputHashMode = "recursive";
  outputHashAlgo = "sha256";
  outputHash = "sha256-iT1pqnnB3Pi2M6lOZKlV4GxIVwA2QXLavXfs5fXsjhM=";
  phases = [ "installPhase" ];
  installPhase = ''
    mkdir -p "$out"
    cd "$out"
    echo "$out" > x
  '';
}

this derivation will be built:
  /nix/store/5pvv7rgxinvp8z66mw4ciry0n9b359rx-test.drv
building '/nix/store/5pvv7rgxinvp8z66mw4ciry0n9b359rx-test.drv'...
installing
/nix/store/5ry50w0v1i4jc0yv911j09pbqhrjvbr2-test

But the resulting store path does not exist after running nix-build, so it is even less obvious that something went wrong:

$ ls /nix/store/5ry50w0v1i4jc0yv911j09pbqhrjvbr2-test
ls: cannot access '/nix/store/5ry50w0v1i4jc0yv911j09pbqhrjvbr2-test': No such file or directory

bergkvist commented 2 years ago

A file does not need to contain the entire output path for the build to fail. In the example above, any of the following contents would have caused a failure:

/nix/store/5ry50w0v1i4jc0yv911j09pbqhrjvbr2-test
TEST5ry50w0v1i4jc0yv911j09pbqhrjvbr2TEST
5ry50w0v1i4jc0yv911j09pbqhrjvbr2

The following contents would have built fine:

5ry50w0v1i4jc0yv911j09pbqhrjvbr
5ry50w0v1i4jc0yv911j09pbqhrjvbr3
ry50w0v1i4jc0yv911j09pbqhrjvbr2
6ry50w0v1i4jc0yv911j09pbqhrjvbr2

So it seems the minimal required character sequence in any file required to cause this problem - is the nix-store hash from the output path.

fgaz commented 2 years ago

@vvs- I get the same with Nix 2.5pre20211007_844dd90 and --allow-import-from-derivation

thufschmitt commented 2 years ago

I think that’s the same issue as https://github.com/NixOS/nix/issues/4859 . Long story short, fixed-output derivations were accidentally allowed to have references, but that was unsound. I believe https://github.com/NixOS/nix/issues/4859#issuecomment-870739762 is the best solution we have, but it needs to be implemented

bergkvist commented 2 years ago

I don't experience the problem if I refer to another previously built derivation - only when something in the output refers to itself. For example, the following builds without any problems:

{ pkgs ? import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/afdb5675a180f347bfa8ae909d4e419fb8b151bd.tar.gz") {}
}: pkgs.stdenv.mkDerivation {
  name = "test";
  outputHashMode = "recursive";
  outputHashAlgo = "sha256";
  outputHash = "1vf422nlhvi2myvpri8j1j9964gjhvkpn2pwqfacss6qzs926s4l";
  phases = [ "installPhase" ];
  installPhase = ''
    mkdir -p "$out"
    cd "$out"
    echo "/nix/store/1z0mfkg7hvdh0yzv0gljxpqplv69b4mk-python3-3.9.4/bin/python3.9" > x
  '';
}

How would this look with the comment from issue 4859? Something like this?

{ pkgs ? import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/afdb5675a180f347bfa8ae909d4e419fb8b151bd.tar.gz") {}
}: pkgs.stdenv.mkDerivation {
  name = "test";
  outputHashMode = "recursive";
  outputHashAlgo = "sha256";
  outputHash = "1vf422nlhvi2myvpri8j1j9964gjhvkpn2pwqfacss6qzs926s4l";
  phases = [ "installPhase" ];
  installPhase = ''
    mkdir -p "$out"
    cd "$out"
    echo "$out" > x
  '';
  outputReferences = [ "$out" ];
}

Is there a reason not to allow self-references? It does mean things are not relocatable - but do you ever want to move something around in /nix/store after it has been built?

kubukoz commented 2 years ago

I'm seeing similar issues. The "shorter repro" in OP's first post gives me an invalid path error as well, but only on 2.4 (not on 2.3.15 - all is fine there).

I have another case of weirdness: a derivation that builds "successfully", but the output path isn't recognized as valid when I try to nix-path-info it:

let
  pkgs = import (
    builtins.fetchTarball {
      url = "https://github.com/nixos/nixpkgs/archive/dc7c656444f0691c5ecd604cc1b4af4b02580645.tar.gz";
      sha256 = "1r6xnackakzrxi8yf37dzz5bjbbb094b7a6ig2pgxjwj241shz02";
    }
  ) {};

  inherit (pkgs) stdenv coursier;

  bloop-coursier-channel = builtins.fetchurl {
    url = "https://github.com/scalacenter/bloop/releases/download/v1.4.11/bloop-coursier.json";
    sha256 = "CoF/1nggjaL17SWmWDcKicfgoyqpOSZUse8f+3TgD0E=";
  };
in

stdenv.mkDerivation rec {
  pname = "bloop-coursier";
  version = "1.4.11";

  platform = if stdenv.isLinux && stdenv.isx86_64 then "x86_64-pc-linux"
  else if stdenv.isDarwin && stdenv.isx86_64 then "x86_64-apple-darwin"
  else throw "unsupported platform";

  dontUnpack = true;
  installPhase = ''
    export COURSIER_CACHE=$(pwd)
    export COURSIER_JVM_CACHE=$(pwd)

    mkdir channel
    cp ${bloop-coursier-channel} channel/bloop.json
    ${coursier}/bin/cs install --install-dir $out --install-platform ${platform} --default-channels=false --channel channel --only-prebuilt=true bloop

    # Remove binary part of the coursier launcher script to make derivation output hash stable
    sed -i '5,$ d' $out/bloop
  '';

  outputHashMode = "recursive";
  outputHashAlgo = "sha256";
  outputHash = if stdenv.isLinux && stdenv.isx86_64 then "01lj7qhj3gjk2g6gxd7997qgahwn4m82ljybvcl1cswx5bm4vjn4"
  else if stdenv.isDarwin && stdenv.isx86_64 then "yv8giwJjgqaSwHRMpk2uhn0rxtPi/bYYtEc4qxAfqXc="
  else throw "unsupported platform";
  fixupPhase = "";
}

I don't know why this is allowed to build at all and doesn't fail with an error like this issue's, but building it again results in a full rebuild (presumably because the out path is not valid). I thought it seemed related. Works on 2.3.15.

Also checked both with --allow-import-from-derivation, didn't help.

thufschmitt commented 2 years ago

Looking at it more closely, this is indeed an instance of #4859 .

To give a bit more details on the issue, content-addressed (CA) paths in Nix were historically not allowed to have references. This is also true in Nix 2.3, but it happens that at some point, fixed-output (FO) derivations seemingly got allowed to have these (meaning that a derivation like the one in the issue description or in #4859 got allowed). But this is semantically wrong because what happens in practice is that the references are discarded before the path gets registered − so as far as Nix is concerned, the output path has no reference, meaning for example that the things it seems to refer to can very-well be garbage-collected.

Some later work (now part of Nix 2.4) allowed content-addressed paths to have references, but to ensure the correctness of the model, these references must be part of the hash part of the store path. Meaning that two content-addressed store paths with the same content on disk, but a different dependency set will effectively be different.

Then the work on content-addressed derivations reworked a bit the logic that was handling the registration of derivation outputs, and fixed the original issue causing FO derivation outputs to have their references dropped. But this also means that their output path can’t really be known in advance (because it depends on the runtime references, which can only be known after the thing has been built), so the derivation isn’t really fixed-output anymore.

And now we had an issue, because all that fixed a real correctness bug, but that got actually (unknowingly) used to work around the fact that CA paths weren’t allowed to hold a reference. So we must find a way restore the previous usages, all while not re-allowing a breach in the model¹.

I don't experience the problem if I refer to another previously built derivation - only when something in the output refers to itself. For example, the following builds without any problems:

The example derivation you give doesn’t have a reference as far as Nix is concerned (because hard-coding a store path won’t make Nix recognize it). If you replace the hard-coded store path by something like ${pkgs.python3}, you’ll get the error again.

How would this look with the comment from issue 4859? Something like this?

Probably yes. Or maybe we could make the outputHash already take into account the references when there’s some (but being careful not to break backwards-compat).

I have another case of weirdness: a derivation that builds "successfully", but the output path isn't recognized as valid when I try to nix-path-info it:

That’s actually the same issue, that just happens to have to symptoms:

If the output path depends on another path, then it will be stored in the “wrong” location (technically the correct one, but not the one that the client expects), hence the invalidity
If the output path depends on itself, then it will also be stored in the “wrong” location, but will hold a reference to its expected location (which is invalid), so the registration will fail because of that.

1: It could also be argued that these were rightfully forbidden because fixed-output derivations should only be as-simple-as-possible fetchers. But given that they exist anyways there’s no much point in arguing about their relevance.

bergkvist commented 2 years ago

Being able to wrap existing fetchers/package managers with something like fixed-output derivations is something I see as very pragmatic. It allows you to get something that works up and running fairly quickly - without having to reimplement fetching logic in nix (which can be a lot of work, and might not work exactly like the original fetcher).

Other package managers often have their own methods for ensuring reproducibility (like package-lock.json in npm). It is nice to be able to outsource the responsibility of reproducibility (as a temporary measure if nothing else).

Maybe nix could have something similar to unsafe in Rust (easily searchable keyword), which allows network access during the build - since this might allow for more incremental adoption of nix for existing projects. I guess fixed output derivations are the closest thing to this right now.

If fixed output derivations are made more strict in terms of what they allow, this will make it harder for existing projects to start using nix. For example if you want to move incrementally from using Dockerfiles to dockerTools - then this might require a lot of upfront work to get going.

vvs- commented 2 years ago

Other package managers often have their own methods for ensuring reproducibility (like package-lock.json in npm). It is nice to be able to outsource the responsibility of reproducibility (as a temporary measure if nothing else).

Seems that you actually want flakes then.

kubukoz commented 2 years ago

Going fully into flakes is a long-term solution, but I don't think users should be forced to go that way if they have existing FODs (at least not with the current state of the union, flakes being experimental and underdocumented).

fgaz commented 2 years ago

Thanks @regnat, that makes sense.

re "complex fetchers" tbh I think they can avoid references too. The only use case i can see for a reference in a FOD is to ensure the hash is up to date (like the rust fetcher does), for example by linking to the lockfile and comparing the link later. But in this case the solution is simple: just copy the lockfile itself in the FOD, or write a hash of its path (since you don't really have to read from it).

Are there other uses for references in FODs that I missed?

bergkvist commented 2 years ago

Example using Python+pip:

{ pkgs ? import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/afdb5675a180f347bfa8ae909d4e419fb8b151bd.tar.gz") {}
}:
let
  python = pkgs.python38;
  setupHook = ''
    export PYTHONPATH="$1/lib/python${python.pythonVersion}/site-packages"
    export PATH="$1/bin"
  '';
in pkgs.stdenv.mkDerivation {
  name = "python-modules";
  buildInputs = [
    python.pkgs.pip
    python.pkgs.setuptools
  ];
  outputHashMode = "recursive";
  outputHash = "sha256-cQuSLO8MovUE45vWlZueUpcVtXl4K2BwZ9/KqCJHm84=";
  phases = [ "installPhase" ];
  installPhase = ''
    mkdir -p "$out" && cd "$out"

    export PIP_PREFIX="$out"
    export PATH="$PIP_PREFIX/bin:$PATH"
    pip install urllib3

    mkdir -p nix-support
    echo "${setupHook}" > nix-support/setup-hook
  '';
}

In this case, pip will create __pycache__-folders with .pyc-files in the library installation directory. These contain precompiled Python bytecode, allowing faster imports/faster startup time. However, these binary files also contain a reference to the original .py-file as an absolute path, which causes a fixed output derivation build to fail.

/nix/store/xvyjgp1qsibfbygdaq2y21hgv6b8k4k9-python-modules/lib/python3.8/site-packages/urllib3/__pycache__/__init__.cpython-38.pyc

contains the string

/nix/store/xvyjgp1qsibfbygdaq2y21hgv6b8k4k9-python-modules/lib/python3.8/site-packages/urllib3/__init__.py

A quick fix to this particular problem is to delete all the __pycache__-folders, which removes the self-references, but results in worse startup times when importing the libraries.

{ pkgs ? import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/afdb5675a180f347bfa8ae909d4e419fb8b151bd.tar.gz") {}
}:
let
  python = pkgs.python38;
  setupHook = ''
    export PYTHONPATH="$1/lib/python${python.pythonVersion}/site-packages"
    export PATH="$1/bin"
  '';
in pkgs.stdenv.mkDerivation {
  name = "python-modules";
  buildInputs = [
    python.pkgs.pip
    python.pkgs.setuptools
  ];
  outputHashMode = "recursive";
  outputHash = "sha256-qbSU0LGtLGix+T4JVp6ZrOX8pKKRIDCt0Ee/3LhMAB8=";
  phases = [ "installPhase" ];
  installPhase = ''
    mkdir -p "$out" && cd "$out"

    export PIP_PREFIX="$out"
    export PATH="$PIP_PREFIX/bin:$PATH"
    pip install urllib3
    rm -rf $(find | grep __pycache__)

    mkdir -p nix-support
    echo "${setupHook}" > nix-support/setup-hook
  '';
}

fgaz commented 2 years ago

I'd argue that the one you posted is not an ideal fetcher, because it builds stuff too and more importantly it saves the built artifacts in $out. I'd split it into a fetching FOD that fetches the packages from pypi using pip download and a normal derivation that builds them.

kubukoz commented 2 years ago

The Bloop example I pasted downloads two files, one of which is a wrapper for the other.

I think I could get away with just keeping the file being wrapped and wrapping it myself (especially since the wrapper is being stripped of binary code), but in general I think non-trivial fetchers like this still have a place, assuming the output is indeed fixed...

bergkvist commented 2 years ago

Why is it bad to save the built artifacts in $out directly?

I agree it might not be ideal to have one derivation that both downloads and builds the packages - but it is a pragmatic/incremental "natural first step" to get something working. At least with pip, since most people using pip will be familiar with pip install ..., but might never have used pip download ... before.

There might be other package managers which doesn't allow the user to independently download and build/install packages like pip does

fgaz commented 2 years ago

Why is it bad to save the built artifacts in $out directly?

because as opposed to the fetched sources they depend on inputs other than the lockfile. For example, the python bytecode may depend on the version of python used to compile it, so the derivation isn't really fixed output.

I agree it might not be ideal to have one derivation that both downloads and builds the packages - but it is a pragmatic/incremental "natural first step" to get something working. At least with pip, since most people using pip will be familiar with pip install ..., but might never have used pip download ... before.

Maybe there could be an escape hatch (disallowed in nixpkgs) that allows to create this kind of package, so that people can still do what you describe as a temporary hack.

There might be other package managers which doesn't allow the user to independently download and build/install packages like pip does

Maven is such a package manager. What I and many others do is:

in a FOD, do a first build, but discard the final artifact and save the downloaded dependencies instead
in a normal derivation, use those dependencies to build the artifact again

This way you can, for example:

apply patches without redownloading all dependencies.
archive the dependencies

Often it's also possible to contribute upstream a download option or command.

Sure, that does require more work, but so does not respecting FHS.

bergkvist commented 2 years ago

because as opposed to the fetched sources they depend on inputs other than the lockfile. For example, the python bytecode may depend on the version of python used to compile it, so the derivation isn't really fixed output.

This can also be the case for pip download (assuming only exact versions are specified, and not hashes) - since it might download binary wheels for a specific platform and Python version. NumPy would be a typical example of this. This can be worked around by having platform/python-version-specific hashes.

That said, pip download also allows for specifying platform and python version as arguments, meaning it should be possible for one platform to produce hashes for all other ones. Doing this with a build is probably harder/less reliable.

Maybe there could be an escape hatch (disallowed in nixpkgs) that allows to create this kind of package, so that people can still do what you describe as a temporary hack.

Yes, this would have been amazing! It would make it is possible to temporarily sacrifice the safety/reproducibility guarantees of nixpkgs to get something up and running quickly. Disallowing it in nixpkgs is a good idea. I guess this could be something like pkgs.stdenv.mkImpureDerivation / pkgs.stdenv.mkUnsafeDerivation. Could also be a flag inside the derivation definition: unsafe = true; or similar.

Something that would be easy for a script/tool to detect, and which makes people using it feel that they are "doing something hacky/wrong" might be ideal. This script/tool could be part of the test suite that runs when someone submits a PR to nixpgks.

nixos-discourse commented 2 years ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-32/19865/1

nixos-discourse commented 2 years ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/differences-in-evaulation-with-fixed-output-dependencies-under-darwin-and-linux/20592/5

milahu commented 1 year ago

I get the (not very clear) error

the error message could be better, similar to https://github.com/NixOS/nix/pull/6585

tomberek commented 10 months ago

Best way forward seems to be to have a way to add things to the store by outputHash as well as specifying references. A similar thing will be needed for RFC92 (dynamic derivations) and likely shares many usecases wrt lang2nix tooling.

expectations: nix 2.3 accepted such FODs without checking them. recommend finding a solution for making such behavior sound rather then removing such checks

nixos-discourse commented 6 months ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/ofborg-pr-error-path-is-not-valid/41507/2

NixOS / nix

"error: path '/nix/store/...' is not valid" when saving a reference to a store path in $out in a fixed output derivation #5509

Context

Example using Python+pip: