search

NixOS / nix

Nix, the purely functional package manager
https://nixos.org/
GNU Lesser General Public License v2.1
11.52k stars 1.44k forks source link

error: path is not valid using nix ==2.18.0 #9052

Open avdv opened 9 months ago

avdv commented 9 months ago

Describe the bug

On CI, we were seeing this problem:

Return code: 1
Error output:
these 2 derivations will be built:
  /nix/store/313azxr6w777rrgz0lcgkp5f0dlm2nvr-posix-toolchain.drv
  /nix/store/qp53d6mv3gs81akp4gv89pk9k7rb76zc-bazel-nixpkgs-posix-toolchain.drv
this path will be fetched (0.09 MiB download, 0.09 MiB unpacked):
  /nix/store/cqam21jchkwyp7k0128g6spcx3mzv6p6-bash-5.1-p16-man
error: path '/nix/store/cqam21jchkwyp7k0128g6spcx3mzv6p6-bash-5.1-p16-man' is not valid

This started to happen around the time when nix 2.18.0 was released. Downgrading to 2.17.0 has resolved this problem.

Steps To Reproduce

  1. clone the daml project: git clone https://github.com/digital-asset/daml.git
  2. cd daml
  3. git checkout 662a4417912df90ba9d9a99817b8ad9703480d06 # not really needed, also fails for commits from 2021
  4. install nix in single-user mode: bash <( curl -sSfL https://nixos.org/nix/install ) --no-daemon
  5. create path_invalid.nix: 
    
with import <nixpkgs> { config = {}; overlays = []; };

let

packages might include lists, e.g. stdenv.initialPath is a list itself,

so we need to flatten packages.

flatten = builtins.concatMap (x: if builtins.isList x then x else [x]); env = buildEnv { name = "posix-toolchain"; paths = flatten [ stdenv.initialPath ]; }; cmd_glob = "${env}/bin/*"; os = if stdenv.isDarwin then "osx" else "linux"; in

runCommand "bazel-nixpkgs-posix-toolchain" { executable = false;

Pointless to do this on a remote machine.

preferLocalBuild = true;
allowSubstitutes = false;

} '' n=$out/nixpkgs_sh_posix.bzl mkdir -p "$(dirname "$n")"

cat >>$n <<EOF
load("@rules_sh//sh:posix.bzl", "posix", "sh_posix_toolchain")
discovered = {
EOF
for cmd in ${cmd_glob}; do
    if [[ -x $cmd ]]; then
        echo "    \'$(basename $cmd)\': \'$cmd\'," >>$n
    fi
done
cat >>$n <<EOF
}
def create_posix_toolchain():
    sh_posix_toolchain(
        name = "nixpkgs_sh_posix",
        cmds = {
            cmd: discovered[cmd]
            for cmd in posix.commands
            if cmd in discovered
        }
    )
EOF

''

6. run:
```bash
DADE_NIXPKGS=$(pwd)/nixpkgs-snapshot

if [[ ! -e "$DADE_NIXPKGS" ]]; then
  echo "Loading outdated or missing nixpkgs snapshot..."
  outpath="$(
    nix-store -Q --realise --indirect --add-root "$DADE_NIXPKGS" \
      "$(nix-instantiate -Q --eval "nix/nixpkgs.nix" -A path \
        | sed 's/^\"//;s/\"$//')"
  )"
  echo "Done loading the nixpkgs snapshot to $outpath"
fi
  1. run: NIX_PATH=nixpkgs=$DADE_NIXPKGS nix-build path_invalid.nix

Expected behavior

I would expect the nix-build command to succeed.

nix-env --version output

$ nix-env --version
nix-env (Nix) 2.18.0

Additional context

I have tracked down the error message to https://github.com/NixOS/nix/blob/57eb62d2307adfd6ce881b41c861eb873153517c/src/libstore/store-api.cc#L727

Priorities

Add :+1: to issues you find important.

Artturin commented 9 months ago

I'm on commit https://github.com/edolstra/nix/commit/3494c295fd14e688cc0451fc2d5d6dd8145ea5c9 in lazy-trees

parent for that merge is https://github.com/edolstra/nix/commit/7f8c99c70c96bf3685e5cad73b38ede801079177 from August 16 and I don't have this issue

Should help with bisecting because 2.17 was released on July 24

git log 7f8c99c70c96bf3685e5cad73b38ede801079177^..2.18.0

Also seen in the #nix-dev:nixos.org room

@trofi said

nix-2.18.0 generates very cryptic errors:

$ nix build --no-link -f nixos system --keep-going
error: path '/nix/store/0mx9wiw18gn44w97jczfg90s2diwz5zp-jq-1.6-dev' is not valid

and warnings:

$ nix build --no-link -f nixos system --keep-going
warning: output lib of input /nix/store/i7vndr6ilzsjdasjvbd9yjvhll365fbd-dconf-0.40.0.drv missing, aborting the resolving
warning: output lib of input /nix/store/i7vndr6ilzsjdasjvbd9yjvhll365fbd-dconf-0.40.0.drv missing, aborting the resolving
warning: output info of input /nix/store/0v0vz7p63cx57wzswsf7m79x1yzmmrd8-findutils-4.9.0.drv missing, aborting the resolving

$ nix build -f. jq.dev made it unstick from error: path '/nix/store/0mx9wiw18gn44w97jczfg90s2diwz5zp-jq-1.6-dev' is not valid.

https://github.com/NixOS/nix/pull/4628 was a big PR so could be that

@Ericson2314 @roberth

garyverhaegen-da commented 9 months ago

If anyone is trying to reproduce, I should add to @avdv's description of the bug that if you happen to already have /nix/store/cqam21jchkwyp7k0128g6spcx3mzv6p6-bash-5.1-p16-man in your Nix store this will succeed, which may make reproduction a bit harder.

I can reliably reproduce the error with Nix 2.18.0 on a new Nix installation, though.

garyverhaegen-da commented 9 months ago

@Artturin I'm one of the maintainers of the https://github.com/digital-asset/daml repo and I can reliably reproduce the bug on 2.18.0 and can confirm it does not appear in with 2.17.0. Are there reasonably-easy-to-follow instructions on how to install arbitrary Nix commits instead of released versions? If so I could help with bisecting.

avdv commented 9 months ago

@Artturin spot on! (oh, it's not from PR #4628, but seems related) It is from https://github.com/NixOS/nix/pull/8829

5e3986f59cb58f48186a49dcec7aa317b4787522 is the first bad commit
commit 5e3986f59cb58f48186a49dcec7aa317b4787522
Author: John Ericson <John.Ericson@Obsidian.Systems>
Date:   Mon Mar 8 16:24:49 2021 -0500

    Adapt scheduler to work with dynamic derivations

    To avoid dealing with an optional `drvPath` (because we might not know
    it yet) everywhere, make an `CreateDerivationAndRealiseGoal`. This goal
    just builds/substitutes the derivation file, and then kicks of a build
    for that obtained derivation; in other words it does the chaining of
    goals when the drv file is missing (as can already be the case) or
    computed (new case).

    This also means the `getDerivation` state can be removed from
    `DerivationGoal`, which makes the `BasicDerivation` / in memory case and
    `Derivation` / drv file file case closer together.

    The map type is factored out for clarity, and because we will soon hvae
    a second use for it (`Derivation` itself).

    Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>

 .../build/create-derivation-and-realise-goal.cc    | 157 +++++++++++++++++++++
 .../build/create-derivation-and-realise-goal.hh    |  96 +++++++++++++
 src/libstore/build/derivation-goal.cc              |  22 +--
 src/libstore/build/derivation-goal.hh              |  15 +-
 src/libstore/build/drv-output-substitution-goal.hh |   4 +-
 src/libstore/build/entry-points.cc                 |  11 +-
 src/libstore/build/goal.cc                         |   2 +-
 src/libstore/build/goal.hh                         |  22 ++-
 src/libstore/build/substitution-goal.hh            |   4 +-
 src/libstore/build/worker.cc                       | 114 ++++++++++++---
 src/libstore/build/worker.hh                       |  22 +++
 src/libstore/derived-path-map.cc                   |  33 +++++
 src/libstore/derived-path-map.hh                   |  73 ++++++++++
 tests/dyn-drv/build-built-drv.sh                   |   4 +-
 14 files changed, 523 insertions(+), 56 deletions(-)
 create mode 100644 src/libstore/build/create-derivation-and-realise-goal.cc
 create mode 100644 src/libstore/build/create-derivation-and-realise-goal.hh
 create mode 100644 src/libstore/derived-path-map.cc
 create mode 100644 src/libstore/derived-path-map.hh
andrevmatos commented 9 months ago

This started affecting my nixos-unstable-small system flake after a nix-collect-garbage and nix flake update, without anything apparent causing it, with a slightly different derivation:

error: path '/nix/store/phscma235mfx30d1dncgizls1rlmh7iz-bash-5.2-p15-dev' is not valid
qbit commented 9 months ago

I was able to get things back up and running nix-store --repair-path /nix/store/...... on the various broken paths and switching nix.package to be pkgs.nixVersions.nix_2_17.

andrevmatos commented 9 months ago

I can confirm nix-store --repair-path on the path above did work around the issue for me (until next gc?); I didn't need to downgrade to nix_2_17, at least while the path is available.

qbit commented 9 months ago

I can confirm nix-store --repair-path on the path above did work around the issue for me (until next gc?); I didn't need to downgrade to nix_2_17, at least while the path is available.

Re the path, are you sure the next one isn't a different bash output? At first I thought it didn't work because there was -info and -man variations.

CobaltCause commented 9 months ago

FWIW, I ran into this with the following store paths:

while upgrading with this commit https://or.computer.surgery/charles/dotfiles/-/commit/7d406812e27e254486ec7a8a01d28bc20eee7694.

My workaround was to run nix copy --from https://cache.nixos.org $affected_store_path for each one until nixos-rebuild worked correctly.

nixos-discourse commented 9 months ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/getting-error-path-not-valid-for-gnugrep-3-11-on-nixos-rebuild-switch/33642/4

nixos-discourse commented 9 months ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/amd-rx-7700-xt-not-being-detected-properly/33683/5

Ericson2314 commented 9 months ago

It would be handy if someone had a way to reproduce that worked with --store so I didn't have to reinstall Nix, build all of NixOS, etc.

I will work on this as the issue is assigned to me, but posting this here in case someone already has one.

Munksgaard commented 9 months ago

I'm seeing the same problem in CI using 2.18.0 (and not locally, using 2.17.0).

garyverhaegen-da commented 9 months ago

In case it helps, here is a slightly smaller reproduction. Works on a machine with a brand new 2.18.0 installation of Nix, and doesn't rely on the daml repo or Bazel. Still suffers from the issue that I can only reproduce it on brand new installs; once the proper packages have been installed from other sources this succeeds. But on a new install it fails. 

EDIT: I can't reproduce this anymore. To avoid confusing people I've commented off the rest of this message.

Ericson2314 commented 9 months ago

@garyverhaegen-da Thanks! I am on the Nix team call but I will try this later. In the meantime would you mind trying with --store /some/directory? (that will bypass any daemon / ambient Nix install.)

garyverhaegen-da commented 9 months ago 
nix-build repro.nix -A '' -I nixpkgs=include/nixpkgs.nix --store $HOME/repro

using the original (i.e. longest) repro.nix from previous post seems to build successfully. To be extra clear, I am not able to reproduce the issue with --store.

Ericson2314 commented 9 months ago

@garyverhaegen-da Oh wow, so when it failed it was also in single user mode global install? It is very odd for --store local (single user mode default) and --store /path/to/dir to have different behavior, as they are almost the same.

Ericson2314 commented 9 months ago

Also Could you test with https://github.com/NixOS/nix/pull/9081 ? I am not sure that reverting that fixed anything cause I haven't reproduced yet!

garyverhaegen-da commented 9 months ago

This is very weird. I can't reproduce anymore, even on brand new machines. I must have screwed something up. I'll get back to you shortly.

Ericson2314 commented 9 months ago

@garyverhaegen-da Other sources of state that perhaps make this non-determinstic is the fetchers catch in ~/.cache.

garyverhaegen-da commented 9 months ago

I must have screwed something up - I'm not able to reproduce using my minimal test case on a brand new machine, though it definitely did work on the machine I was testing with. I'll get back to you as soon as I have a new minimal that works on new machines. And I'll retry --store then.

Ericson2314 commented 9 months ago

@garyverhaegen-da No worries, good luck, thank you for doing this it helps me out greatly :)

garyverhaegen-da commented 9 months ago

Alright, I have something again, tried on two new machines so this time I'm reasonably confident. It's way less "minimal" than the previous one though. Here are the steps, on a fresh machine with Nix 2.18.0 and direnv installed:

  1. git clone https://github.com/digital-asset/daml.git (I'd really like to get rid of this but I'm not there yet.)
  2. cd daml
  3. direnv allow
  4. Add the files below.
  5. From within the daml folder, with direnv enabled, nix-build repro.nix -A '' -I include/nixpkgs.nix.

Files to add are as follows:

daml/repro.nix

with import <nixpkgs> { config = {}; overlays = []; };

let
  flatten = builtins.concatMap (x: if builtins.isList x then x else [x]);
  env = buildEnv {
    name = "posix-toolchain";
    paths = flatten [ stdenv.initialPath ];
  };
  cmd_glob = "${env}/bin/*";
  os = if stdenv.isDarwin then "osx" else "linux";
in

runCommand "bazel-nixpkgs-posix-toolchain"
  { executable = false;
    # Pointless to do this on a remote machine.
    preferLocalBuild = true;
    allowSubstitutes = false;
  }
  ''
    n=$out/nixpkgs_sh_posix.bzl
    mkdir -p "$(dirname "$n")"

    cat >>$n <<EOF
    load("@rules_sh//sh:posix.bzl", "posix", "sh_posix_toolchain")
    discovered = {
    EOF
    for cmd in ${cmd_glob}; do
        if [[ -x $cmd ]]; then
            echo "    '$(basename $cmd)': '$cmd'," >>$n
        fi
    done
    cat >>$n <<EOF
    }
    def create_posix_toolchain():
        sh_posix_toolchain(
            name = "nixpkgs_sh_posix",
            cmds = {
                cmd: discovered[cmd]
                for cmd in posix.commands
                if cmd in discovered
            }
        )
    EOF
  ''

daml/include/nixpkgs.nix

# Pinned version of nixpkgs that we use for our development and deployment.

{ system ? import ./system.nix
, ...
}:

let
  # See ./nixpkgs/README.md for upgrade instructions.
  src = import ./nixpkgs;

  # package overrides
  overrides = _: pkgs: rec {
    nodejs = pkgs.nodejs-16_x;
    nodejs14 = pkgs.nodejs-14_x;
    ephemeralpg = pkgs.ephemeralpg.overrideAttrs(oldAttrs: {
      installPhase = ''
        mkdir -p $out
        PREFIX=$out make install
        wrapProgram $out/bin/pg_tmp --prefix PATH : ${pkgs.postgresql_11}/bin:$out/bin
      '';
    });
    scala_2_13 = pkgs.scala_2_13.overrideAttrs (oldAttrs: rec {
      version = "2.13.10";
      name = "scala-2.13.10";
      src = pkgs.fetchurl {
        url = "https://www.scala-lang.org/files/archive/${name}.tgz";
        sha256 = "sha256-sBRhWZzQeGBCxktTN5D0XlG6u5HFLcRl2EaDjpcBnMQ=";
      };
    });

    bazel_5 = pkgs.bazel_5.overrideAttrs(oldAttrs: {
      patches = oldAttrs.patches ++ [
        # This has been upstreamed but it's only available from Bazel 7.0.0-pre.20230215.2
        # (maybe we can get it backported?)
        # https://github.com/bazelbuild/bazel/commit/6115d94cd05864fe5c6e5f774e9482b3b4976976
        ./bazel-retry-cache.patch
      ];
    });
    haskell = pkgs.haskell // {
      compiler = pkgs.haskell.compiler // {
        ghc902 =
          if system == "aarch64-darwin" then
            pkgs.haskell.compiler.ghc902.override(oldAttrs: {
              buildTargetLlvmPackages = pkgs.llvmPackages_12;
              llvmPackages = pkgs.llvmPackages_12;
            })
          else
            pkgs.haskell.compiler.ghc902;
      };
    };
  };

  nixpkgs = import src {
    inherit system;

    # pin the overlays
    overlays = [overrides];

    config.allowUnfree = true;
    config.allowBroken = true;
  };
in
  nixpkgs

daml/include/system.nix

builtins.currentSystem

daml/include/nixpkgs/default.nix

let
  spec = builtins.fromJSON (builtins.readFile ./default.src.json);
  src = builtins.fetchTarball {
    url = "https://github.com/${spec.owner}/${spec.repo}/archive/${spec.rev}.tar.gz";
    sha256 = spec.sha256;
  };
in
  src

daml/include/nixpkgs/default.src.json

{
  "owner": "NixOS",
  "repo": "nixpkgs",
  "rev": "9a82a9b5248919805a2400266ebd881d5783df2a",
  "sha256": "142x1zq3cjadgmvfv0paydlq268pfinllqpq2vl0vxwdiq2nr9iz"
}

To make it a bit easier, I've made a branch on the daml repo called debug-nix-2.1.8. I've confirmed the following steps work (i.e. reproduce the bug, i.e. fail) on a new machine with direnv and Nix 2.18.0:

  1. git clone https://github.com/digital-asset/daml.git
  2. cd daml
  3. git checkout 744ef2903053abdbd2e9edf35393c9102f16a439 (current tip of debug-nix-2.18, adds the five files above)
  4. direnv allow - there's unfortunately quite a lot that happens here in the daml repo
  5. nix-build repro.nix -A '' -I nixpkgs=include/nixpkgs.nix

This is the smallest repro I've got at this point. At least we've eliminated Bazel from the equation. I'll try to get this smaller tomorrow.

If anyone else looks into this in the meantime, I think the most likely variable (compared to running the same repro.nix with the same include folder outside of the daml repo, which works) is the dev-env/etc/nix.conf file, though all of dev-env/ seems plausible.

Note that following these steps but replacing the last step with

nix-build repro.nix -A '' -I nixpkgs=include/nixpkgs.nix --store $HOME/nix-tmp

yields a successful build, i.e. doesn't reproduce the issue. This may point to the core issue being something that happens when the Nix store is initialized in the nix install, or when it is pre-populated by direnv allow. I'm not sure yet how to investigate which of those it might be.

nixos-discourse commented 9 months ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-09-29-nix-team-meeting-minutes-90/33774/1

nixos-discourse commented 9 months ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-10-02-nix-team-meeting-minutes-91/33775/1

googleson78 commented 9 months ago

Also hit this, also for bash, gnugrep and findutils. Repairing the paths and switching to 2.17 seems to have fixed the issue at this stage of my build.

garyverhaegen-da commented 9 months ago

I can still reliably reproduce on Nix 2.18.0, but I am not able to reproduce on Nix 2.18.1. I'm still working on reducing the repro.

Artturin commented 9 months ago

Updating nix in nixpkgs to 2.18.1 https://github.com/NixOS/nixpkgs/pull/258839

Ericson2314 commented 9 months ago

@garyverhaegen-da instead of doing --store ..., one can also do export NIX_CONFIG= store = .... This had the advantage of being propogated to all commands (unless something overwrites it), including the direnv one. I hope that would make the custom store version finally repro the issue.

garyverhaegen-da commented 9 months ago

I have to stop for the day, but here's where I'm at: I've managed to completely isolate the problem from the daml repo. Here is a script that reproduces the issue for me.

The ad-hoc.sh command is an internal tool that creates GCP VMs, which do not have Nix installed but do have a writeable-to-all /nix folder, which is why we can do a single-user Nix install with no sudo in the script below.

#!/usr/bin/env bash

set -euo pipefail

DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
cd "$DIR"

./ad-hoc.sh create ubuntu
source .envrc.private
trap "./ad-hoc.sh destroy $MACHINE" EXIT

ssh="ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=$(mktemp)"

while ! $ssh gary@$IP "echo connected" 2>/dev/null; do sleep 1; done

($ssh gary@$IP <<'EOF'
bash <(curl -sSfL https://releases.nixos.org/nix/nix-2.18.0/install)
source $HOME/.nix-profile/etc/profile.d/nix.sh

cat <<'SEMVER' >semver.nix
let
  system = builtins.currentSystem;
  src = builtins.fetchTarball {
    url = "https://github.com/NixOS/nixpkgs/archive/9a82a9b5248919805a2400266ebd881d5783df2a.tar.gz";
    sha256 = "142x1zq3cjadgmvfv0paydlq268pfinllqpq2vl0vxwdiq2nr9iz";
  };
  pkgs = import src {
    inherit system;

    config.allowUnfree = true;
    config.allowBroken = true;
  };
in {
  semver = pkgs.callPackage (
{ lib, stdenv, fetchFromGitHub }:

stdenv.mkDerivation rec {
  version = "7cd86658";
  name = "semver-tool-${version}";

  src = fetchFromGitHub {
    inherit name;
    owner = "fsaintjacques";
    repo = "semver-tool";
    rev = version;
    sha256 = "1v70dgp5yl4di90p8gzbj97zylgc9q971ds5g84id78c2fh3xh28";
  };

  phases = [ "unpackPhase" "installPhase" ];

  installPhase =
    ''
      mkdir -p $out/bin
      cp $src/src/semver $out/bin/
      patchShebangs $out/bin/semver
    '';

  meta = {
    homepage = https://github.com/fsaintjacques/semver-tool;
    description = "The semver shell utility";
    license = lib.licenses.gpl3;
    platforms = lib.platforms.all;
  };
}) {};
}
SEMVER

cat <<'REPRO' > repro.nix
with import <nixpkgs> { config = {}; overlays = []; };

let
  flatten = builtins.concatMap (x: if builtins.isList x then x else [x]);
  env = buildEnv {
    name = "posix-toolchain";
    paths = flatten [ stdenv.initialPath ];
  };
  cmd_glob = "${env}/bin/*";
  os = if stdenv.isDarwin then "osx" else "linux";
in

runCommand "bazel-nixpkgs-posix-toolchain"
  { executable = false;
    # Pointless to do this on a remote machine.
    preferLocalBuild = true;
    allowSubstitutes = false;
  }
  ''
    n=$out/nixpkgs_sh_posix.bzl
    mkdir -p "$(dirname "$n")"

    cat >>$n <<EOF
    load("@rules_sh//sh:posix.bzl", "posix", "sh_posix_toolchain")
    discovered = {
    EOF
    for cmd in ${cmd_glob}; do
        if [[ -x $cmd ]]; then
            echo "    '$(basename $cmd)': '$cmd'," >>$n
        fi
    done
    cat >>$n <<EOF
    }
    def create_posix_toolchain():
        sh_posix_toolchain(
            name = "nixpkgs_sh_posix",
            cmds = {
                cmd: discovered[cmd]
                for cmd in posix.commands
                if cmd in discovered
            }
        )
    EOF
  ''
REPRO

mkdir -p include/nixpkgs

cat <<'NIXPKGS' > include/nixpkgs.nix
# Pinned version of nixpkgs that we use for our development and deployment.

{ system ? import ./system.nix
, ...
}:

let
  # See ./nixpkgs/README.md for upgrade instructions.
  src = import ./nixpkgs;

  # package overrides
  overrides = _: pkgs: rec {
    nodejs = pkgs.nodejs-16_x;
    nodejs14 = pkgs.nodejs-14_x;
    ephemeralpg = pkgs.ephemeralpg.overrideAttrs(oldAttrs: {
      installPhase = ''
        mkdir -p $out
        PREFIX=$out make install
        wrapProgram $out/bin/pg_tmp --prefix PATH : ${pkgs.postgresql_11}/bin:$out/bin
      '';
    });
    scala_2_13 = pkgs.scala_2_13.overrideAttrs (oldAttrs: rec {
      version = "2.13.10";
      name = "scala-2.13.10";
      src = pkgs.fetchurl {
        url = "https://www.scala-lang.org/files/archive/${name}.tgz";
        sha256 = "sha256-sBRhWZzQeGBCxktTN5D0XlG6u5HFLcRl2EaDjpcBnMQ=";
      };
    });

    bazel_5 = pkgs.bazel_5.overrideAttrs(oldAttrs: {
      patches = oldAttrs.patches ++ [
        # This has been upstreamed but it's only available from Bazel 7.0.0-pre.20230215.2
        # (maybe we can get it backported?)
        # https://github.com/bazelbuild/bazel/commit/6115d94cd05864fe5c6e5f774e9482b3b4976976
        ./bazel-retry-cache.patch
      ];
    });
    haskell = pkgs.haskell // {
      compiler = pkgs.haskell.compiler // {
        ghc902 =
          if system == "aarch64-darwin" then
            pkgs.haskell.compiler.ghc902.override(oldAttrs: {
              buildTargetLlvmPackages = pkgs.llvmPackages_12;
              llvmPackages = pkgs.llvmPackages_12;
            })
          else
            pkgs.haskell.compiler.ghc902;
      };
    };
  };

  nixpkgs = import src {
    inherit system;

    # pin the overlays
    overlays = [overrides];

    config.allowUnfree = true;
    config.allowBroken = true;
  };
in
  nixpkgs
NIXPKGS

cat <<'SYSTEM' > include/system.nix
builtins.currentSystem
SYSTEM

cat <<'DEFAULT_NIX' > include/nixpkgs/default.nix
let
  spec = builtins.fromJSON (builtins.readFile ./default.src.json);
  src = builtins.fetchTarball {
    url = "https://github.com/${spec.owner}/${spec.repo}/archive/${spec.rev}.tar.gz";
    sha256 = spec.sha256;
  };
in
  src
DEFAULT_NIX

cat <<'DEFAULT_JSON' > include/nixpkgs/default.src.json
{
  "owner": "NixOS",
  "repo": "nixpkgs",
  "rev": "9a82a9b5248919805a2400266ebd881d5783df2a",
  "sha256": "142x1zq3cjadgmvfv0paydlq268pfinllqpq2vl0vxwdiq2nr9iz"
}
DEFAULT_JSON

#export NIX_STORE_DIR=$(mktemp -d)
nix-build semver.nix -A "semver"
nix-build repro.nix -A '' -I nixpkgs=include/nixpkgs.nix
EOF
) 2>&1 | grep -oP 'path .* is not valid'

I suspect the Nix code can be further reduced/simplified but I'm out of time for today.

Note that uncommenting the line

#export NIX_STORE_DIR=$(mktemp -d)

makes it take a lot longer (a few minutes with the default store, about 2h on these machines with the custom store), and fails to reproduce the issue (i.e. the build succeeds).

Ericson2314 commented 9 months ago

Thanks again @garyverhaegen-da for all your hard work.

Note that NIX_STORE_DIR has a different effect (it won't chroot internally so you are still using /nix/store within builds and thus can substitute prebuilt binaries) as opposed to the NIX_CONFIG thing which will do the chroot and thus shouldn't take so long.

Ericson2314 commented 9 months ago

@garyverhaegen-da Also, I would be happy to get on a call with you about this, if that might help.

garyverhaegen-da commented 9 months ago

Thanks again @garyverhaegen-da for all your hard work.

Note that NIX_STORE_DIR has a different effect (it won't chroot internally so you are still using /nix/store within builds and thus can substitute prebuilt binaries) as opposed to the NIX_CONFIG thing which will do the chroot and thus shouldn't take so long.

Can you give me the exact syntax for the NIX_CONFIG option? I'm not very familiar with Nix and the naive export NIX_CONFIG=store=$(mktemp -d) resulted in an error.

garyverhaegen-da commented 9 months ago

I think this is the smallest I can make it:

#!/usr/bin/env bash

set -euo pipefail

DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
cd "$DIR"

./ad-hoc.sh create ubuntu
source .envrc.private
trap "./ad-hoc.sh destroy $MACHINE" EXIT

ssh="ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=$(mktemp)"

while ! $ssh gary@$IP "echo connected" 2>/dev/null; do sleep 1; done

($ssh gary@$IP <<'EOF'
bash <(curl -sSfL https://releases.nixos.org/nix/nix-2.18.0/install)
source $HOME/.nix-profile/etc/profile.d/nix.sh

cat <<'REPRO' > repro.nix
let
  system = builtins.currentSystem;
  src = builtins.fetchTarball {
    url = "https://github.com/NixOS/nixpkgs/archive/9a82a9b5248919805a2400266ebd881d5783df2a.tar.gz";
    sha256 = "142x1zq3cjadgmvfv0paydlq268pfinllqpq2vl0vxwdiq2nr9iz";
  };
  pkgs = import src {};
  flatten = builtins.concatMap (x: if builtins.isList x then x else [x]);
  env = pkgs.buildEnv {
    name = "posix-toolchain";
    paths = flatten [ pkgs.stdenv.initialPath ];
  };
  cmd_glob = "${env}/bin/*";
in
{
  semver = pkgs.callPackage (
{ lib, stdenv, fetchFromGitHub }:

stdenv.mkDerivation rec {
  version = "7cd86658";
  name = "semver-tool-${version}";
  src = fetchFromGitHub {
    inherit name;
    owner = "fsaintjacques";
    repo = "semver-tool";
    rev = version;
    sha256 = "1v70dgp5yl4di90p8gzbj97zylgc9q971ds5g84id78c2fh3xh28";
  };
  phases = [ "unpackPhase" "installPhase" ];
  installPhase =
    ''
      mkdir -p $out/bin
      cp $src/src/semver $out/bin/
      patchShebangs $out/bin/semver
    '';
}) {};
repro = pkgs.runCommand "repro" {}
  ''
    echo ${cmd_glob}
  '';
}
REPRO

nix-build repro.nix -A "semver"
nix-build repro.nix -A 'repro'
EOF
) 2>&1 | grep -oP 'path .* is not valid'

At this point everything I can think of either fails in another way or succeeds.

solidsnakedev commented 6 months ago

same issue here, I was able to fix it as follows:

❯ sudo nixos-rebuild switch --flake .
warning: Git tree '/home/homeserver/nixos-config' is dirty
building the system configuration...
warning: Git tree '/home/homeserver/nixos-config' is dirty
error: path '/nix/store/wjizx94ksgs39haljg9pn2yinr5ic9az-bash-interactive-5.2-p21-doc' is not valid

❯ nix copy --from https://cache.nixos.org /nix/store/wjizx94ksgs39haljg9pn2yinr5ic9az-bash-interactive-5.2-p21-doc

❯ sudo nixos-rebuild switch --flake .
warning: Git tree '/home/homeserver/nixos-config' is dirty
building the system configuration...
warning: Git tree '/home/homeserver/nixos-config' is dirty
error: path '/nix/store/1bv944c4z7f3d62j462i184aj5lv9nqn-bash-interactive-5.2-p21-info' is not valid

❯ nix copy --from https://cache.nixos.org /nix/store/1bv944c4z7f3d62j462i184aj5lv9nqn-bash-interactive-5.2-p21-info

❯ sudo nixos-rebuild switch --flake .
warning: Git tree '/home/homeserver/nixos-config' is dirty
building the system configuration...
warning: Git tree '/home/homeserver/nixos-config' is dirty
error: path '/nix/store/74jgj1g3sab05jyp7hyppp777cpzjgs3-gnugrep-3.11-info' is not valid

❯ nix copy --from https://cache.nixos.org /nix/store/74jgj1g3sab05jyp7hyppp777cpzjgs3-gnugrep-3.11-info

❯ sudo nixos-rebuild switch --flake .
warning: Git tree '/home/homeserver/nixos-config' is dirty
building the system configuration...
warning: Git tree '/home/homeserver/nixos-config' is dirty
error: path '/nix/store/xg2z6k8zjm2r4xyk8mzycpz09nxv2klc-findutils-4.9.0-info' is not valid

❯ nix copy --from https://cache.nixos.org /nix/store/xg2z6k8zjm2r4xyk8mzycpz09nxv2klc-findutils-4.9.0-info
roberth commented 6 months ago

If you want to

a-h commented 6 months ago

I have another reproduction of what I believe to be this issue, on Nix 2.18.1.

In https://github.com/a-h/nix-copy I've got scripts to set up two NixOS VMs - source and target based on the same configuration.nix. The configuration.nix enables SSH, and sets a password for the user adrian of password: https://github.com/a-h/nix-copy/blob/main/configuration.nix

The flake.nix in the repo has a devShell which includes libvirt for running the virtual machines, the xc tool I use to simplify running complex commands, and the virsh-json tool I wrote to parse tables that are output from virsh commands and turn them into JSON.

The first command is xc build-iso which builds an ISO from the configuration.nix.

virt-run-all then runs two copies of the ISO using virt-install. One is called source, and the other is called target.

Once the VMs are stared, the virt-ssh command returns the IP addresses of the source and target machine, then I could run the tests.

Export package to disk

First, I SSH onto the source machine, and copy the hello command to disk:

nix copy --to file://$PWD/hello nixpkgs#hello --extra-experimental-features nix-command --extra-experimental-features flakes

Copy from source to target using scp

Next, I copy the results via SCP to the target machine, which mimics the behaviour of using a USB drive etc.

[adrian@nixos:~]$ scp -r ./hello adrian@192.168.122.182:/home/adrian/hello                                 
The authenticity of host '192.168.122.182 (192.168.122.182)' can't be established.
ED25519 key fingerprint is SHA256:O8ltjoGolRWGEFxQV4dK+RktPvTt347gqV8s/QfKA+0.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.122.182' (ED25519) to the list of known hosts.
(adrian@192.168.122.182) Password: 
ji01n9vinnj22nbrb86nx8a1ssgpilx8.narinfo                                                                                                                                           100%  606   581.0KB/s   00:00    
i93s7xxblavsacpy82zdbn4kplsyq48l.narinfo                                                                                                                                           100%  567     5.6MB/s   00:00    
9y8pmvk8gdwwznmkzxa6pwyah52xy3nk.narinfo                                                                                                                                           100%  659     7.2MB/s   00:00    
ldrslljw4rg026nw06gyrdwl78k77vyq.narinfo                                                                                                                                           100%  513     5.6MB/s   00:00    
zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm.narinfo                                                                                                                                           100%  600     6.7MB/s   00:00    
12116yvlqh0sszlf6ipagkivxgh4ybn15d7qynvl34k87bmwkyv6.nar.xz                                                                                                                        100%   49KB  95.4MB/s   00:00    
168qxfl2rswcwbpvajr43r7s2fxan4lk69kyf7wasffga1ava9g8.nar.xz                                                                                                                        100% 6357KB 100.9MB/s   00:00    
0crc324g4mk9qv2kawr3w0rklvix854m7iarg8z848pibm3sxjfv.nar.xz                                                                                                                        100%   87KB 197.6MB/s   00:00    
1qi9kgysh7rpfnj20zxiys6xg15265ws0sc5kybmw1zdj8asc37j.nar.xz                                                                                                                        100%   50KB 198.3MB/s   00:00    
1rm2gqcly43l1mnd7iwyhslcfxnlg91y5c7y49y8dww1qwl0lsyp.nar.xz                                                                                                                        100%  402KB 366.1MB/s   00:00    
nix-cache-info                                                                                                                                                                     100%   21   206.8KB/s   00:00

Get the store path on the source machine

With the data copied to the target machine, I can print out the path. For some reason, there's no newline.

[adrian@nixos:~]$ nix path-info nixpkgs#hello --extra-experimental-features nix-command --extra-experimental-features flakes
evaluating derivation 'flake:nixpkgs#hello'/nix/store/zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-hello-2.12.1

SSH into target machine and import the results of nix copy

Next, I exit the connection with source and SSH into target.

[adrian@nixos:~]$ exit
logout
Connection to 192.168.122.175 closed.
(nix:nix-shell-env) bash-5.2$ ssh adrian@192.168.122.182
(adrian@192.168.122.182) Password:

Then, I try to run the nix copy operation to add the SCP'd files to the store.

[adrian@nixos:~]$ nix copy --all --no-check-sigs --from file://$PWD/hello --extra-experimental-features nix-command --extra-experimental-features flakes
error: path '/nix/store/ji01n9vinnj22nbrb86nx8a1ssgpilx8-libidn2-2.3.4' is not valid

Got unexpected error

As seen, I got the "error: path '/nix/store/ji01n9vinnj22nbrb86nx8a1ssgpilx8-libidn2-2.3.4' is not valid".

Why does SSH work?

If I log on to source and use ssh as the target, then it works OK:

ssh adrian@192.168.122.175
[adrian@nixos:~]$ nix copy --to ssh-ng://192.168.122.182 nixpkgs#hello --extra-experimental-features nix-command --extra-experimental-features flakes
(adrian@192.168.122.182) Password:

Then... on the target...

(nix:nix-shell-env) bash-5.2$ ssh adrian@192.168.122.182
(adrian@192.168.122.182) Password: 
Last login: Wed Jan 10 12:57:10 2024 from 192.168.122.1

[adrian@nixos:~]$ nix shell /nix/store/zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-hello-2.12.1 --extra-experimental-features nix-command --extra-experimental-features flakes

[adrian@nixos:~]$ hello
Hello, world!

So, it's interesting that the nix copy operation seems to work if the target is SSH, but doesn't work if the target is a file path.

Code

https://github.com/a-h/nix-copy

a-h commented 6 months ago

I gave @solidsnakedev's workaround a try to help debug the issue further.

I noticed that it is possible to download from the cache to disk, and restore from disk on the target machine. So, I reproduced the issue, having copied the package from source to target. Running on target, I can't add the files to the store:

[adrian@nixos:~]$ nix copy --all --no-check-sigs --from file://$PWD/hello --extra-experimental-features nix-command --extra-experimental-features flakes                                                             
error: path '/nix/store/ldrslljw4rg026nw06gyrdwl78k77vyq-xgcc-12.3.0-libgcc' is not valid

But if I download it from the Nix cache to disk...

[adrian@nixos:~]$ nix copy --from https://cache.nixos.org /nix/store/ldrslljw4rg026nw06gyrdwl78k77vyq-xgcc-12.3.0-libgcc --to file://$PWD/libgcc --extra-experimental-features nix-command

Then from disk into the cache...

[adrian@nixos:~]$ nix copy --all --no-check-sigs --from file://$PWD/libgcc --extra-experimental-features nix-command --extra-experimental-features flakes

The operation succeeds:

[adrian@nixos:~]$ nix copy --all --no-check-sigs --from file://$PWD/hello --extra-experimental-features nix-command --extra-experimental-features flakes

Was the narinfo or NAR file corrupted during transfer?

My first thought was that perhaps the scp operation didn't work, so I compared the narinfo files, and the one received from the source machine was identical to the one downloaded from the Nix cache on the local machine.

adrian@nixos:~]$ cat hello/ldrslljw4rg026nw06gyrdwl78k77vyq.narinfo                                                                                                                                                 
StorePath: /nix/store/ldrslljw4rg026nw06gyrdwl78k77vyq-xgcc-12.3.0-libgcc                                                                                                                                            
URL: nar/1qi9kgysh7rpfnj20zxiys6xg15265ws0sc5kybmw1zdj8asc37j.nar.xz                                                                                                                                                 
Compression: xz                                                                                                                                                                                                      
FileHash: sha256:1qi9kgysh7rpfnj20zxiys6xg15265ws0sc5kybmw1zdj8asc37j                                                                                                                                                
FileSize: 50868                                                                                                                                                                                                      
NarHash: sha256:18agdkqz0vzfxznc4h8ss7ql483lyjzjng185xcrwmdx7x1vc9a8                                                                                                                                                 
NarSize: 142456                                                                                                                                                                                                      
References:                                                                                                                                                                                                          
Deriver: m8hpbqa0vsnl1nr6g1n8bg792qakap68-xgcc-12.3.0.drv                                                                                                                                                            
Sig: cache.nixos.org-1:v+YiK82YiH+LXgSu9bPFbhPH1KYc190XB7Jb1c8TOvOaN8EsvDsDNcmCOxdLgR6YaB9N0LTp3uIa6ctQrdf/DA==                                                                                                      

[adrian@nixos:~]$ cat libgcc/ldrslljw4rg026nw06gyrdwl78k77vyq.narinfo                                                                                                                                                
StorePath: /nix/store/ldrslljw4rg026nw06gyrdwl78k77vyq-xgcc-12.3.0-libgcc                                                                                                                                            
URL: nar/1qi9kgysh7rpfnj20zxiys6xg15265ws0sc5kybmw1zdj8asc37j.nar.xz                                                                                                                                                 
Compression: xz                                                                                                                                                                                                      
FileHash: sha256:1qi9kgysh7rpfnj20zxiys6xg15265ws0sc5kybmw1zdj8asc37j                                                                                                                                                
FileSize: 50868                                                                                                                                                                                                      
NarHash: sha256:18agdkqz0vzfxznc4h8ss7ql483lyjzjng185xcrwmdx7x1vc9a8                                                                                                                                                 
NarSize: 142456                                                                                                                                                                                                      
References:                                                                                                                                                                                                          
Deriver: m8hpbqa0vsnl1nr6g1n8bg792qakap68-xgcc-12.3.0.drv                                                                                                                                                            
Sig: cache.nixos.org-1:v+YiK82YiH+LXgSu9bPFbhPH1KYc190XB7Jb1c8TOvOaN8EsvDsDNcmCOxdLgR6YaB9N0LTp3uIa6ctQrdf/DA==

In addition, the NAR files themselves are identical. 

[adrian@nixos:~]$ sha256sum libgcc/nar/1qi9kgysh7rpfnj20zxiys6xg15265ws0sc5kybmw1zdj8asc37j.nar.xz                                                                                                                   
f20ca61592ed075e979f8569a07931a284d78df6b17f20a475371fa8fd9b29e2  libgcc/nar/1qi9kgysh7rpfnj20zxiys6xg15265ws0sc5kybmw1zdj8asc37j.nar.xz                                                                             

[adrian@nixos:~]$ sha256sum hello/nar/1qi9kgysh7rpfnj20zxiys6xg15265ws0sc5kybmw1zdj8asc37j.nar.xz                                                                                                                    
f20ca61592ed075e979f8569a07931a284d78df6b17f20a475371fa8fd9b29e2  hello/nar/1qi9kgysh7rpfnj20zxiys6xg15265ws0sc5kybmw1zdj8asc37j.nar.xz

I think this shows that the problem doesn't seem to be caused by the download or transfer of the archives.

So what else could be causing this?

a-h commented 6 months ago

Reproducible bisection

Now that I have a reproducible testing process, I noticed that the nixpkgs version was changing between each run, and that I was using a recent commit, so I tried with nixpkgs/23.11 and it worked perfectly well, whereas with the latest unstable nixpkgs it doesn't.

[adrian@nixos:~]$ nix copy --to file://$PWD/hello github:NixOS/nixpkgs/23.11#hello --extra-experimental-features nix-command --extra-experimental-features flakes

Since it was working in 23.11, and not in the latest master branch, I decided to hunt it down. I got all the commits since 23.11 on the main branch with git log --oneline | sed '/Release NixOS 23.11/q' and bisected them.

There had been 13,269 commits since the latest, so I had to do 13 checks to find the exact commit when it started to break.

# 76090aacf4b6 - broken
# 7b2399a63c27 - working
# 140f2db977a0 - working
# 468a6bab44bf - broken
# 44f2f5ce5aaf - broken
# f9480bd35d76 - broken
# b122013b2373 - broken
# 0c909de8e6ab - broken
# 5902643e53f2 - broken
# a1e9171ca3c1 - broken
# 73b3a1450f4a - broken
# 67fc0e51da63 - broken
# 5621fb9e2dc5 - working

So I tracked it down to here. 

67fc0e51da63    R. Ryantm   Fri Dec 15 05:07:22 2023 +0000  biome: 1.4.0 -> 1.4.1
Broken ^
5621fb9e2dc5    Nikolay Korotkiy    Tue Dec 12 02:04:18 2023 +0400  mosquitto: fix pkg-config files
Working ^

Why on earth would https://github.com/NixOS/nixpkgs/commit/67fc0e51da63 cause a nix copy operation of hello to print error: path '/nix/store/ji01n9vinnj22nbrb86nx8a1ssgpilx8-libidn2-2.3.4' is not valid?

a-h commented 5 months ago

Reproduction in Nix 2.19.2

Since the workaround was apparently to use Nix 2.18.1, and I was able to reproduce the issue in it, I decided to check if it had been fixed in a subsequent version of Nix accidentally.

I updated the reproduction repo to use 2.19.2 in https://github.com/a-h/nix-copy/commit/e855ecb8cf523e8cc975def4efafdbefd9ba6a3f and then created two new virtual machines - 192.168.122.153 (source) and 192.168.122.152 (target).

0 0 /home/adrian-hesketh/github.com/a-h/nix-copy % ssh adrian@192.168.122.153
The authenticity of host '192.168.122.153 (192.168.122.153)' can't be established.
ED25519 key fingerprint is SHA256:5s8NaYrdeH3MZhUKKrExqw44JmKwS+q9/h8cKkoiFtI.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.122.153' (ED25519) to the list of known hosts.
(adrian@192.168.122.153) Password: 

[adrian@nixos:~]$ nix --version
nix (Nix) 2.19.2

[adrian@nixos:~]$ nix copy --to file://$PWD/hello github:NixOS/nixpkgs/67fc0e51da63#hello

[adrian@nixos:~]$ scp -r ./hello/ adrian@192.168.122.152:/home/adrian/hello
The authenticity of host '192.168.122.152 (192.168.122.152)' can't be established.
ED25519 key fingerprint is SHA256:92Q0nHQWTKDy/DdFcCjKgOo8rDSlRNwjAyaoG47a00g.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.122.152' (ED25519) to the list of known hosts.
(adrian@192.168.122.152) Password: 
1rm2gqcly43l1mnd7iwyhslcfxnlg91y5c7y49y8dww1qwl0lsyp.nar.xz                                                                                                                        100%  402KB 112.7MB/s   00:00    
12116yvlqh0sszlf6ipagkivxgh4ybn15d7qynvl34k87bmwkyv6.nar.xz                                                                                                                        100%   49KB 179.4MB/s   00:00    
0crc324g4mk9qv2kawr3w0rklvix854m7iarg8z848pibm3sxjfv.nar.xz                                                                                                                        100%   87KB 170.6MB/s   00:00    
168qxfl2rswcwbpvajr43r7s2fxan4lk69kyf7wasffga1ava9g8.nar.xz                                                                                                                        100% 6357KB  67.6MB/s   00:00    
1qi9kgysh7rpfnj20zxiys6xg15265ws0sc5kybmw1zdj8asc37j.nar.xz                                                                                                                        100%   50KB 136.2MB/s   00:00    
ldrslljw4rg026nw06gyrdwl78k77vyq.narinfo                                                                                                                                           100%  513     4.5MB/s   00:00    
zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm.narinfo                                                                                                                                           100%  600     5.6MB/s   00:00    
i93s7xxblavsacpy82zdbn4kplsyq48l.narinfo                                                                                                                                           100%  567     5.6MB/s   00:00    
ji01n9vinnj22nbrb86nx8a1ssgpilx8.narinfo                                                                                                                                           100%  606     5.9MB/s   00:00    
nix-cache-info                                                                                                                                                                     100%   21   192.7KB/s   00:00    
9y8pmvk8gdwwznmkzxa6pwyah52xy3nk.narinfo                                                                                                                                           100%  659     7.1MB/s   00:00    

[adrian@nixos:~]$ exit
logout
Connection to 192.168.122.153 closed.
0 0 /home/adrian-hesketh/github.com/a-h/nix-copy % ssh adrian@192.168.122.152
The authenticity of host '192.168.122.152 (192.168.122.152)' can't be established.
ED25519 key fingerprint is SHA256:92Q0nHQWTKDy/DdFcCjKgOo8rDSlRNwjAyaoG47a00g.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.122.152' (ED25519) to the list of known hosts.
(adrian@192.168.122.152) Password: 

[adrian@nixos:~]$ nix copy --all --no-check-sigs --from file://$PWD/hello
error: path '/nix/store/ji01n9vinnj22nbrb86nx8a1ssgpilx8-libidn2-2.3.4' is not valid

[adrian@nixos:~]$ nix --version
nix (Nix) 2.19.2

[adrian@nixos:~]$
a-h commented 5 months ago

@roberth - I would appreciate your thoughts on how to proceed. @'ing you because it says to in your profile. 😁

But the commit doesn't seem to make sense as the cause - I can't see how a minor change to a Rust package would result in error: path '/nix/store/ji01n9vinnj22nbrb86nx8a1ssgpilx8-libidn2-2.3.4' is not valid

Any ideas?

nixos-discourse commented 5 months ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tiny-nixos-virtualbox-appliance/38145/3

a-h commented 5 months ago

Reading through the comments above, I noticed that there was a suggestion that it worked in Nix 2.17, but it doesn't.

[adrian@nixos:~]$ nix --version                
nix (Nix) 2.17.1                               

[adrian@nixos:~]$ nix copy --all --verbose --no-check-sigs --from file://$PWD/hello                        
copying 5 paths...                                                                                        
copying path '/nix/store/9y8pmvk8gdwwznmkzxa6pwyah52xy3nk-x' from 'file:///home/adrian/hello'...
copying path '/nix/store/i93s7xxblavsacpy82zdbn4kplsyq48l-x' from 'file:///home/adrian/hello'...
copying path '/nix/store/ji01n9vinnj22nbrb86nx8a1ssgpilx8-x' from 'file:///home/adrian/hello'...
copying path '/nix/store/ldrslljw4rg026nw06gyrdwl78k77vyq-x' from 'file:///home/adrian/hello'...
copying path '/nix/store/zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-x' from 'file:///home/adrian/hello'...
error: path '/nix/store/ji01n9vinnj22nbrb86nx8a1ssgpilx8-libidn2-2.3.4' is not valid
a-h commented 5 months ago

The Nix copy output does contain everything required to restore the store, but it probably isn't doing it in the right order.

The store that's being copied from the source passes verification.

[adrian@nixos:~]$ nix store verify -vvvvv --store "file://$PWD/hello" --all
querying info about '/nix/store/9y8pmvk8gdwwznmkzxa6pwyah52xy3nk-x' on 'file:///home/adrian/hello'...
querying info about '/nix/store/i93s7xxblavsacpy82zdbn4kplsyq48l-x' on 'file:///home/adrian/hello'...
querying info about '/nix/store/ji01n9vinnj22nbrb86nx8a1ssgpilx8-x' on 'file:///home/adrian/hello'...
querying info about '/nix/store/kg7q3p9zy4q6spr8a41nd8njni8h8ayg-x' on 'file:///home/adrian/hello'...
querying info about '/nix/store/ldrslljw4rg026nw06gyrdwl78k77vyq-x' on 'file:///home/adrian/hello'...
querying info about '/nix/store/zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-x' on 'file:///home/adrian/hello'...
starting pool of 0 threads
checking '/nix/store/9y8pmvk8gdwwznmkzxa6pwyah52xy3nk-glibc-2.38-27'...
querying info about '/nix/store/9y8pmvk8gdwwznmkzxa6pwyah52xy3nk-glibc-2.38-27' on 'file:///home/adrian/hello'...
checking '/nix/store/i93s7xxblavsacpy82zdbn4kplsyq48l-libunistring-1.1'...
querying info about '/nix/store/i93s7xxblavsacpy82zdbn4kplsyq48l-libunistring-1.1' on 'file:///home/adrian/hello'...
checking '/nix/store/ji01n9vinnj22nbrb86nx8a1ssgpilx8-libidn2-2.3.4'...
querying info about '/nix/store/ji01n9vinnj22nbrb86nx8a1ssgpilx8-libidn2-2.3.4' on 'file:///home/adrian/hello'...
checking '/nix/store/kg7q3p9zy4q6spr8a41nd8njni8h8ayg-source'...
querying info about '/nix/store/kg7q3p9zy4q6spr8a41nd8njni8h8ayg-source' on 'file:///home/adrian/hello'...
checking '/nix/store/ldrslljw4rg026nw06gyrdwl78k77vyq-xgcc-12.3.0-libgcc'...
querying info about '/nix/store/ldrslljw4rg026nw06gyrdwl78k77vyq-xgcc-12.3.0-libgcc' on 'file:///home/adrian/hello'...
checking '/nix/store/zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-hello-2.12.1'...
querying info about '/nix/store/zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-hello-2.12.1' on 'file:///home/adrian/hello'...

And, the hello check passes:

nix store verify --store "file://$PWD/hello" /nix/store/zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-hello-2.12.1

Including recursively.

[adrian@nixos:~]$ nix store verify -vvvvv --recursive --store "file://$PWD/hello" /nix/store/zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-hello-2.12.1
querying info about missing paths...
starting pool of 24 threads
querying info about '/nix/store/zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-hello-2.12.1' on 'file:///home/adrian/hello'...
querying info about '/nix/store/9y8pmvk8gdwwznmkzxa6pwyah52xy3nk-glibc-2.38-27' on 'file:///home/adrian/hello'...
querying info about '/nix/store/ji01n9vinnj22nbrb86nx8a1ssgpilx8-libidn2-2.3.4' on 'file:///home/adrian/hello'...
querying info about '/nix/store/i93s7xxblavsacpy82zdbn4kplsyq48l-libunistring-1.1' on 'file:///home/adrian/hello'...
querying info about '/nix/store/ldrslljw4rg026nw06gyrdwl78k77vyq-xgcc-12.3.0-libgcc' on 'file:///home/adrian/hello'...
starting pool of 0 threads
checking '/nix/store/i93s7xxblavsacpy82zdbn4kplsyq48l-libunistring-1.1'...
checking '/nix/store/ji01n9vinnj22nbrb86nx8a1ssgpilx8-libidn2-2.3.4'...
checking '/nix/store/ldrslljw4rg026nw06gyrdwl78k77vyq-xgcc-12.3.0-libgcc'...
checking '/nix/store/9y8pmvk8gdwwznmkzxa6pwyah52xy3nk-glibc-2.38-27'...
checking '/nix/store/zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-hello-2.12.1'...

Here's the graph of packages produced by nix-store -q --graph /nix/store/zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-hello-2.12.1/bin/hello:

image

From the graph, it's clear that the order of installation must be libunistring-1.1, then libidn2-2.3.4 and xgcc-12.3.0-libgcc can happen at the same time, followed by glibc-2.38-27, and then finally hello-2.12.1.

This is exactly the checking order I see in the logs above. However, nix copy doesn't do the same order.

[adrian@nixos:~]$ nix copy -vvvvv --all --no-check-sigs --from file://$PWD/hello                          
performing daemon worker op: 31                                                                                                                                                                                      
copying 5 paths...                                                                                        
querying info about '/nix/store/9y8pmvk8gdwwznmkzxa6pwyah52xy3nk-glibc-2.38-27' on 'file:///home/adrian/hello'...     
querying info about '/nix/store/i93s7xxblavsacpy82zdbn4kplsyq48l-libunistring-1.1' on 'file:///home/adrian/hello'...                                                                                                                
querying info about '/nix/store/ji01n9vinnj22nbrb86nx8a1ssgpilx8-libidn2-2.3.4' on 'file:///home/adrian/hello'...                                                                                                                
querying info about '/nix/store/ldrslljw4rg026nw06gyrdwl78k77vyq-xgcc-12.3.0-libgcc' on 'file:///home/adrian/hello'...                                                                                                                
querying info about '/nix/store/zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-hello-2.12.1' on 'file:///home/adrian/hello'...     
copying path '/nix/store/9y8pmvk8gdwwznmkzxa6pwyah52xy3nk-glibc-2.38-27' from 'file:///home/adrian/hello'...                                                                                                                     
performing daemon worker op: 44                                                                                                                                                                                      
acquiring write lock on '/nix/var/nix/temproots/3145'                                                                                                                                                                
locking path '/nix/store/9y8pmvk8gdwwznmkzxa6pwyah52xy3nk-glibc-2.38-27'                                  
lock acquired on '/nix/store/9y8pmvk8gdwwznmkzxa6pwyah52xy3nk-glibc-2.38-27.lock'                                                                                                                                    
copying path '/nix/store/i93s7xxblavsacpy82zdbn4kplsyq48l-libunistring-1.1' from 'file:///home/adrian/hello'...                                                                                                                     
lock released on '/nix/store/9y8pmvk8gdwwznmkzxa6pwyah52xy3nk-glibc-2.38-27.lock'                         
copying path '/nix/store/ji01n9vinnj22nbrb86nx8a1ssgpilx8-libidn2-2.3.4' from 'file:///home/adrian/hello'...                                                                                                                     
copying path '/nix/store/ldrslljw4rg026nw06gyrdwl78k77vyq-xgcc-12.3.0-libgcc' from 'file:///home/adrian/hello'...                                                                                                                     
copying path '/nix/store/zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-hello-2.12.1' from 'file:///home/adrian/hello'...          
error: path '/nix/store/ji01n9vinnj22nbrb86nx8a1ssgpilx8-libidn2-2.3.4' is not valid

The original logs aren't very helpful, because instead of zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-hello-2.12.1 they have zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-x, so I did some text edits to the logs above to add the package name instead of -x.

I don't know what it's supposed to be doing, but it looks like it's releasing the glibc lock too early to me. Shouldn't it be holding the lock until all dependencies of glibc (are copied, i.e. libunistring-1.1, libidn2-2.3.4 and xgcc-12.3.0-libgcc)?

Reading through https://github.com/NixOS/nix/blob/e6e160a0758c0354ed810b0ec9846ed885adcc11/src/libstore/store-api.cc#L1087-L1169, it says it is doing a toplogical sort, but the copy operation looks like it's working through the alphabetic ordering of the paths, instead of following the dependency graph order (as the recursive verification does).

Does that help at all?

a-h commented 5 months ago

Since all of the paths are actually available on disk, you can copy them one at a time with a shell script without having to go to the Internet to fetch them again with this (assumes nix-export is the folder containing the results of nix copy to a filesystem):

for x in `grep StorePath nix-export/*.narinfo | awk '{print $2}'`; do nix copy $x --from file://$PWD/nix-export/ --offline; done