search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
17.99k stars 14.01k forks source link

ociTools.buildContainer erroring with "path [foo] is not in the Nix store" #316652

Open tomfitzhenry opened 4 months ago

tomfitzhenry commented 4 months ago

Describe the bug

pkgs.ociTools.buildContainer is failing with errors such as error: path '/nix/store/306znyj77fv49kwnkpxmb0j2znqpa8bj-bash-5.2p26/bin/bash' is not in the Nix store.

This example in the documentation ( https://nixos.org/manual/nixpkgs/unstable/#ssec-pkgs-ociTools-buildContainer-examples ) fails in this way, as well as when I try to integrate this into my NixOS configuration.

Steps To Reproduce

Follow https://nixos.org/manual/nixpkgs/unstable/#ssec-pkgs-ociTools-buildContainer-examples , e.g.

$ cat default.nix
{ ociTools, lib, bash }:
ociTools.buildContainer {
  args = [
    (lib.getExe bash)
  ];

  readonly = false;
}

$ nix-build --expr '(import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/ad57eef4ef0659193044870c731987a6df5cf56b.tar.gz") {}).callPackage ./default.nix {}'
these 3 derivations will be built:
  /nix/store/18fgd6sz9q8d9r9xwbxkz4nid8kpq2ah-runtime-deps.drv
  /nix/store/5vjjr5sxibh9r7llxs2n03qzjn4r6p56-config.json.drv
  /nix/store/1v6mjrp8nj9a1xns1i75r3zdxzab2kqp-join.drv
building '/nix/store/5vjjr5sxibh9r7llxs2n03qzjn4r6p56-config.json.drv'...
error: path '/nix/store/306znyj77fv49kwnkpxmb0j2znqpa8bj-bash-5.2p26/bin/bash' is not in the Nix store

Expected behavior

I expected an OCI image to be built.

Notify maintainers

@ShamrockLee @aib

Add a :+1: reaction to issues you find important.

ShamrockLee commented 4 months ago

Thanks for pinning, but I haven't worked on ociTools before and am unsure if I could help.

ociTools.buildContainer sounds fantastic, and I hope to eventually get it to work with Apptainer and SingularityCE (the container packages I'm maintaining).

Cc: @SomeoneSerge (who might also be interested)

ShamrockLee commented 4 months ago

I cannot reproduce this issue on my laptop.

Reproducer:

nix-repl> lib = import ./lib                                                                                  
nix-repl> pkgs = import ./. { config = { allowAliases = false; checkMeta = true; }; }                         

nix-repl> oci-example = pkgs.ociTools.buildContainer { args = [ (lib.getExe pkgs.bash) ]; readonly = false; } 

nix-repl> oci-example
«derivation /nix/store/hcw309zyix75sym0mhz8hn2kfi8kx4sw-join.drv»
$ nix-build --no-out-link /nix/store/hcw309zyix75sym0mhz8hn2kfi8kx4sw-join.drv
aib commented 4 months ago

I... am not sure what's going on.

I stopped my nix adventure when I was still deeply confused by numerous similar-but-different concepts despite completing multiple tutorials and spending several days practicing. As such, I don't know what nix-build is doing, or how it's related to default.nix or the repl output in the examples above.

I would love to help if I could do so in any way, though. (I'm even willing to go through another tutorial, if anyone has a good one handy.)

ShamrockLee commented 4 months ago

Update:

I tried using an inlined version of @tomfitzhenry's reproducer and saw the same error.

nix-build --expr '(import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/ad57eef4ef0659193044870c731987a6df5cf56b.tar.gz") { }).callPackage ({ ociTools, lib, bash }: ociTools.buildContainer { args = [ (lib.getExe bash) ]; readonly = false; }) { }'

Something is wrong, but I wonder why the two reproducers would behave differently.

aib commented 4 months ago

Looks like it's due to the writeReferencesToFile -> writeClosure change in e9fd4389d2cc2289150d067623c908870ee556e3 . I'll continue applying my non-nix-specific skills to try to debug further...

aib commented 4 months ago

This is probably obvious for others, but:

writeReferencesToFile's command takes the file graph:

/nix/store/4vzal97iq3dmrgycj8r0gflrh51p8w1s-bash-5.2p26

2
/nix/store/4vzal97iq3dmrgycj8r0gflrh51p8w1s-bash-5.2p26
/nix/store/ksk3rnb0ljx8gngzk19jlmbjyvac4hw6-glibc-2.38-44
/nix/store/d8w5qfswmgxcjqwnmqw2v9r8amrvdlpb-xgcc-13.2.0-libgcc

0
/nix/store/krqp9wj3rgalmqv04y0sqw987mxsnddn-libidn2-2.3.7

2
/nix/store/krqp9wj3rgalmqv04y0sqw987mxsnddn-libidn2-2.3.7
/nix/store/vqvbn2z8wyrjwvayjb2vy5krhh1kis9b-libunistring-1.1
/nix/store/ksk3rnb0ljx8gngzk19jlmbjyvac4hw6-glibc-2.38-44

3
/nix/store/d8w5qfswmgxcjqwnmqw2v9r8amrvdlpb-xgcc-13.2.0-libgcc
/nix/store/krqp9wj3rgalmqv04y0sqw987mxsnddn-libidn2-2.3.7
/nix/store/ksk3rnb0ljx8gngzk19jlmbjyvac4hw6-glibc-2.38-44
/nix/store/vqvbn2z8wyrjwvayjb2vy5krhh1kis9b-libunistring-1.1

1
/nix/store/vqvbn2z8wyrjwvayjb2vy5krhh1kis9b-libunistring-1.1

And converts it to the file $out:

/nix/store/4vzal97iq3dmrgycj8r0gflrh51p8w1s-bash-5.2p26
/nix/store/d8w5qfswmgxcjqwnmqw2v9r8amrvdlpb-xgcc-13.2.0-libgcc
/nix/store/krqp9wj3rgalmqv04y0sqw987mxsnddn-libidn2-2.3.7
/nix/store/ksk3rnb0ljx8gngzk19jlmbjyvac4hw6-glibc-2.38-44
/nix/store/vqvbn2z8wyrjwvayjb2vy5krhh1kis9b-libunistring-1.1

writeClosure uses $NIX_ATTRS_JSON_FILE and jq to do the same. The problem is, runCommand doesn't like the "${pkgs.bash}/bin/bash" passed in exportReferencesGraph.graph.

(I guess I'm taking notes here in this comment)

ShamrockLee commented 4 months ago

Looks like it's due to the writeReferencesToFile -> writeClosure change in https://github.com/NixOS/nixpkgs/commit/e9fd4389d2cc2289150d067623c908870ee556e3 .

I'm sorry for any trouble the change might cause. I was the author of this change. The corresponding PR is #178717.

writeReferencesToFile takes a graph:

It took me quite some time to understand when working on the writeReferencesToFile -> writeClosure refactoring. To simplify things, writeClosure utilizes the new, JSON-based exportReferencesGraph interface when specifying __structuredAttrs = true. Here is the discussion thread: https://github.com/NixOS/nixpkgs/pull/178717#discussion_r1518978226

aib commented 4 months ago

One thing I don't understand is why we get the "x is not in the Nix store" error when x is an actual file under /nix/store.

As for buildContainer, are we sure we want to be parsing args as a list of dependencies? The first element is a command, for sure, but how do we know the rest are filenames, let alone files that need to exist and be copied into the rootfs?

I think we need a separate "dependencies" parameter where pkgs.bash can go. The Docker builder seems to have... copyToRoot? Is that what a derivation means?

ShamrockLee commented 4 months ago

It turns out that the samples (tests.trivial-builders.references.samples) of the test case tests.trivial-builders.references doesn't cover lib.getExe-related use cases. The error appears as soon as such a sample is added.

ShamrockLee commented 4 months ago

One thing I don't understand is why we get the "x is not in the Nix store" error when x is an actual file under /nix/store.

I think it's a bug in the structured exportReferencesGraph interface of Nix. We can reproduce it simply by using the interface:

Legacy interface, outPath, successful build

nix-build --no-out-link -E "let lib = import ./lib; pkgs = import ./. { }; path = pkgs.hello.outPath; in pkgs.runCommandLocal ''reference-graph-legacy'' { exportReferencesGraph = [ ''graph'' path ]; } ''cat ./graph | tee \$out''"

this derivation will be built:
  /nix/store/diigmg6bqxpixcj3dzzqavmc6783pjs1-reference-graph-legacy.drv
building '/nix/store/diigmg6bqxpixcj3dzzqavmc6783pjs1-reference-graph-legacy.drv'...
/nix/store/1q9vc0lq7qjlfjz47mfmlzdf86c543jy-xgcc-13.2.0-libgcc

0
/nix/store/ic63ay0py10fyryaw7345k4ps32da33w-libidn2-2.3.7

2
/nix/store/ic63ay0py10fyryaw7345k4ps32da33w-libidn2-2.3.7
/nix/store/yfp7dr8m7zi7kxk49wd714gwvhb105hf-libunistring-1.1
/nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52

3
/nix/store/1q9vc0lq7qjlfjz47mfmlzdf86c543jy-xgcc-13.2.0-libgcc
/nix/store/ic63ay0py10fyryaw7345k4ps32da33w-libidn2-2.3.7
/nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52
/nix/store/yfp7dr8m7zi7kxk49wd714gwvhb105hf-libunistring-1.1

1
/nix/store/yfp7dr8m7zi7kxk49wd714gwvhb105hf-libunistring-1.1
/nix/store/yh6x9ia5kxxw3w90vkb09vqgfvhb416k-hello-2.12.1

2
/nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52
/nix/store/yh6x9ia5kxxw3w90vkb09vqgfvhb416k-hello-2.12.1
/nix/store/ik4m89xg09jx21fkjh54varbj450m5dp-reference-graph-legacy

Legacy interface, getExe, successful build

ix-build --no-out-link -E "let lib = import ./lib; pkgs = import ./. { }; path = lib.getExe pkgs.hello; in pkgs.runCommandLocal ''reference-graph-legacy'' { exportReferencesGraph = [ ''graph'' path ]; } ''cat ./graph | tee \$out''"

this derivation will be built:
  /nix/store/rlzp3ilxh7xghsn22vnjblccijhg7c29-reference-graph-legacy.drv
building '/nix/store/rlzp3ilxh7xghsn22vnjblccijhg7c29-reference-graph-legacy.drv'...
/nix/store/1q9vc0lq7qjlfjz47mfmlzdf86c543jy-xgcc-13.2.0-libgcc

0
/nix/store/ic63ay0py10fyryaw7345k4ps32da33w-libidn2-2.3.7

2
/nix/store/ic63ay0py10fyryaw7345k4ps32da33w-libidn2-2.3.7
/nix/store/yfp7dr8m7zi7kxk49wd714gwvhb105hf-libunistring-1.1
/nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52

3
/nix/store/1q9vc0lq7qjlfjz47mfmlzdf86c543jy-xgcc-13.2.0-libgcc
/nix/store/ic63ay0py10fyryaw7345k4ps32da33w-libidn2-2.3.7
/nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52
/nix/store/yfp7dr8m7zi7kxk49wd714gwvhb105hf-libunistring-1.1

1
/nix/store/yfp7dr8m7zi7kxk49wd714gwvhb105hf-libunistring-1.1
/nix/store/yh6x9ia5kxxw3w90vkb09vqgfvhb416k-hello-2.12.1

2
/nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52
/nix/store/yh6x9ia5kxxw3w90vkb09vqgfvhb416k-hello-2.12.1
/nix/store/h9bigga3j2ah1dpakjz1bpsyzy6iqvf2-reference-graph-legacy

Structured interface, outPath, successful build

nix-build --no-out-link -E "let lib = import ./lib; pkgs = import ./. { }; path = pkgs.hello.outPath; in pkgs.runCommandLocal ''reference-graph-structured'' { __structuredAttrs = true; nativeBuildInputs = [ pkgs.buildPackages.jq ]; exportReferencesGraph.graph = [ path ]; } ''jq . \$NIX_ATTRS_JSON_FILE | tee \$out''"

this derivation will be built:
  /nix/store/6wrl680daaffrm8hy4ahlkslv7dlq28q-reference-graph-structured.drv
building '/nix/store/6wrl680daaffrm8hy4ahlkslv7dlq28q-reference-graph-structured.drv'...
structuredAttrs is enabled
{
  "allowSubstitutes": false,
  "buildCommand": "jq . $NIX_ATTRS_JSON_FILE | tee $out",
  "buildInputs": [],
  "builder": "/nix/store/306znyj77fv49kwnkpxmb0j2znqpa8bj-bash-5.2p26/bin/bash",
  "cmakeFlags": [],
  "configureFlags": [],
  "depsBuildBuild": [],
  "depsBuildBuildPropagated": [],
  "depsBuildTarget": [],
  "depsBuildTargetPropagated": [],
  "depsHostHost": [],
  "depsHostHostPropagated": [],
  "depsTargetTarget": [],
  "depsTargetTargetPropagated": [],
  "doCheck": false,
  "doInstallCheck": false,
  "enableParallelBuilding": true,
  "enableParallelChecking": true,
  "enableParallelInstalling": true,
  "env": {},
  "exportReferencesGraph": {
    "graph": [
      "/nix/store/yh6x9ia5kxxw3w90vkb09vqgfvhb416k-hello-2.12.1"
    ]
  },
  "graph": [
    {
      "closureSize": 159560,
      "narHash": "sha256:10q8iyvfmpfck3yiisnj1j8vp6lq3km17r26sr95zpdf9mgmk69s",
      "narSize": 159560,
      "path": "/nix/store/1q9vc0lq7qjlfjz47mfmlzdf86c543jy-xgcc-13.2.0-libgcc",
      "references": [],
      "valid": true
    },
    {
      "closureSize": 2190120,
      "narHash": "sha256:0mfvcix1h7prq9wwpwfbwyj56pkdm3w05i69zwvd911fg27w7ccy",
      "narSize": 361136,
      "path": "/nix/store/ic63ay0py10fyryaw7345k4ps32da33w-libidn2-2.3.7",
      "references": [
        "/nix/store/ic63ay0py10fyryaw7345k4ps32da33w-libidn2-2.3.7",
        "/nix/store/yfp7dr8m7zi7kxk49wd714gwvhb105hf-libunistring-1.1"
      ],
      "valid": true
    },
    {
      "closureSize": 32473824,
      "narHash": "sha256:1ljsk162rvyhjdsfbiiq26lsm8ahz68y1g366r6wwbq31d1aanah",
      "narSize": 30124144,
      "path": "/nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52",
      "references": [
        "/nix/store/1q9vc0lq7qjlfjz47mfmlzdf86c543jy-xgcc-13.2.0-libgcc",
        "/nix/store/ic63ay0py10fyryaw7345k4ps32da33w-libidn2-2.3.7",
        "/nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52"
      ],
      "valid": true
    },
    {
      "closureSize": 1828984,
      "narHash": "sha256:1akas0jc72xbc8fz1a8ja1gk5ghh1jqm9s1p46p67sg2bcjfiwzg",
      "narSize": 1828984,
      "path": "/nix/store/yfp7dr8m7zi7kxk49wd714gwvhb105hf-libunistring-1.1",
      "references": [
        "/nix/store/yfp7dr8m7zi7kxk49wd714gwvhb105hf-libunistring-1.1"
      ],
      "valid": true
    },
    {
      "closureSize": 32700384,
      "narHash": "sha256:1s8ax2azcifcnxsn0ny1gk75hd6hfmm3l5nd56xpsxmya2gwnyg9",
      "narSize": 226560,
      "path": "/nix/store/yh6x9ia5kxxw3w90vkb09vqgfvhb416k-hello-2.12.1",
      "references": [
        "/nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52",
        "/nix/store/yh6x9ia5kxxw3w90vkb09vqgfvhb416k-hello-2.12.1"
      ],
      "valid": true
    }
  ],
  "mesonFlags": [],
  "name": "reference-graph-structured",
  "nativeBuildInputs": [
    "/nix/store/n5chc8i5za5jrr4biwwgx7a96p0a7aq9-jq-1.7.1-dev"
  ],
  "outputs": {
    "out": "/nix/store/l30rfsn0lcbnxrd248s40ak22rvrqkm3-reference-graph-structured"
  },
  "passAsFile": [
    "buildCommand"
  ],
  "patches": [],
  "preferLocalBuild": true,
  "propagatedBuildInputs": [],
  "propagatedNativeBuildInputs": [],
  "stdenv": "/nix/store/rg6agzlb4gcl4w2126m16qm4mgr8jgi6-stdenv-linux",
  "strictDeps": false,
  "system": "x86_64-linux"
}
/nix/store/l30rfsn0lcbnxrd248s40ak22rvrqkm3-reference-graph-structured

Structured interface, getExe, failed build

nix-build --no-out-link -E "let lib = import ./lib; pkgs = import ./. { }; path = lib.getExe pkgs.hello; in pkgs.runCommandLocal ''reference-graph-structured'' { __structuredAttrs = true; nativeBuildInputs = [ pkgs.buildPackages.jq ]; exportReferencesGraph.graph = [ path ]; } ''jq . \$NIX_ATTRS_JSON_FILE | tee \$out''"

this derivation will be built:
  /nix/store/n4a18ikia9x5pa5x4hsjvsgb1gnv66wh-reference-graph-structured.drv
error: path '/nix/store/yh6x9ia5kxxw3w90vkb09vqgfvhb416k-hello-2.12.1/bin/hello' is not in the Nix store
ShamrockLee commented 4 months ago

The upstream seems to have noticed this. For details, see NixOS/nix#10549 and NixOS/nix#10565.

aib commented 4 months ago

Ahh great catch! Thanks.

I still think it will be a good idea to have a dedicated "dependencies" or "copyToRoot" field for this, given args is a command array.