Open tomfitzhenry opened 4 months ago
Thanks for pinning, but I haven't worked on ociTools
before and am unsure if I could help.
ociTools.buildContainer
sounds fantastic, and I hope to eventually get it to work with Apptainer and SingularityCE (the container packages I'm maintaining).
Cc: @SomeoneSerge (who might also be interested)
I cannot reproduce this issue on my laptop.
Reproducer:
nix-repl> lib = import ./lib
nix-repl> pkgs = import ./. { config = { allowAliases = false; checkMeta = true; }; }
nix-repl> oci-example = pkgs.ociTools.buildContainer { args = [ (lib.getExe pkgs.bash) ]; readonly = false; }
nix-repl> oci-example
«derivation /nix/store/hcw309zyix75sym0mhz8hn2kfi8kx4sw-join.drv»
$ nix-build --no-out-link /nix/store/hcw309zyix75sym0mhz8hn2kfi8kx4sw-join.drv
I... am not sure what's going on.
I stopped my nix adventure when I was still deeply confused by numerous similar-but-different concepts despite completing multiple tutorials and spending several days practicing. As such, I don't know what nix-build is doing, or how it's related to default.nix or the repl output in the examples above.
I would love to help if I could do so in any way, though. (I'm even willing to go through another tutorial, if anyone has a good one handy.)
Update:
I tried using an inlined version of @tomfitzhenry's reproducer and saw the same error.
nix-build --expr '(import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/ad57eef4ef0659193044870c731987a6df5cf56b.tar.gz") { }).callPackage ({ ociTools, lib, bash }: ociTools.buildContainer { args = [ (lib.getExe bash) ]; readonly = false; }) { }'
Something is wrong, but I wonder why the two reproducers would behave differently.
Looks like it's due to the writeReferencesToFile -> writeClosure change in e9fd4389d2cc2289150d067623c908870ee556e3 . I'll continue applying my non-nix-specific skills to try to debug further...
This is probably obvious for others, but:
writeReferencesToFile
's command takes the file graph
:
/nix/store/4vzal97iq3dmrgycj8r0gflrh51p8w1s-bash-5.2p26
2
/nix/store/4vzal97iq3dmrgycj8r0gflrh51p8w1s-bash-5.2p26
/nix/store/ksk3rnb0ljx8gngzk19jlmbjyvac4hw6-glibc-2.38-44
/nix/store/d8w5qfswmgxcjqwnmqw2v9r8amrvdlpb-xgcc-13.2.0-libgcc
0
/nix/store/krqp9wj3rgalmqv04y0sqw987mxsnddn-libidn2-2.3.7
2
/nix/store/krqp9wj3rgalmqv04y0sqw987mxsnddn-libidn2-2.3.7
/nix/store/vqvbn2z8wyrjwvayjb2vy5krhh1kis9b-libunistring-1.1
/nix/store/ksk3rnb0ljx8gngzk19jlmbjyvac4hw6-glibc-2.38-44
3
/nix/store/d8w5qfswmgxcjqwnmqw2v9r8amrvdlpb-xgcc-13.2.0-libgcc
/nix/store/krqp9wj3rgalmqv04y0sqw987mxsnddn-libidn2-2.3.7
/nix/store/ksk3rnb0ljx8gngzk19jlmbjyvac4hw6-glibc-2.38-44
/nix/store/vqvbn2z8wyrjwvayjb2vy5krhh1kis9b-libunistring-1.1
1
/nix/store/vqvbn2z8wyrjwvayjb2vy5krhh1kis9b-libunistring-1.1
And converts it to the file $out
:
/nix/store/4vzal97iq3dmrgycj8r0gflrh51p8w1s-bash-5.2p26
/nix/store/d8w5qfswmgxcjqwnmqw2v9r8amrvdlpb-xgcc-13.2.0-libgcc
/nix/store/krqp9wj3rgalmqv04y0sqw987mxsnddn-libidn2-2.3.7
/nix/store/ksk3rnb0ljx8gngzk19jlmbjyvac4hw6-glibc-2.38-44
/nix/store/vqvbn2z8wyrjwvayjb2vy5krhh1kis9b-libunistring-1.1
writeClosure uses $NIX_ATTRS_JSON_FILE
and jq
to do the same. The problem is, runCommand
doesn't like the "${pkgs.bash}/bin/bash"
passed in exportReferencesGraph.graph
.
(I guess I'm taking notes here in this comment)
Looks like it's due to the writeReferencesToFile -> writeClosure change in https://github.com/NixOS/nixpkgs/commit/e9fd4389d2cc2289150d067623c908870ee556e3 .
I'm sorry for any trouble the change might cause. I was the author of this change. The corresponding PR is #178717.
writeReferencesToFile takes a graph:
It took me quite some time to understand when working on the writeReferencesToFile -> writeClosure refactoring. To simplify things, writeClosure
utilizes the new, JSON-based exportReferencesGraph
interface when specifying __structuredAttrs = true
. Here is the discussion thread: https://github.com/NixOS/nixpkgs/pull/178717#discussion_r1518978226
One thing I don't understand is why we get the "x is not in the Nix store" error when x is an actual file under /nix/store.
As for buildContainer, are we sure we want to be parsing args
as a list of dependencies? The first element is a command, for sure, but how do we know the rest are filenames, let alone files that need to exist and be copied into the rootfs?
I think we need a separate "dependencies" parameter where pkgs.bash
can go. The Docker builder seems to have... copyToRoot
? Is that what a derivation means?
It turns out that the samples (tests.trivial-builders.references.samples
) of the test case tests.trivial-builders.references
doesn't cover lib.getExe
-related use cases. The error appears as soon as such a sample is added.
One thing I don't understand is why we get the "x is not in the Nix store" error when x is an actual file under /nix/store.
I think it's a bug in the structured exportReferencesGraph
interface of Nix. We can reproduce it simply by using the interface:
outPath
, successful buildnix-build --no-out-link -E "let lib = import ./lib; pkgs = import ./. { }; path = pkgs.hello.outPath; in pkgs.runCommandLocal ''reference-graph-legacy'' { exportReferencesGraph = [ ''graph'' path ]; } ''cat ./graph | tee \$out''"
this derivation will be built:
/nix/store/diigmg6bqxpixcj3dzzqavmc6783pjs1-reference-graph-legacy.drv
building '/nix/store/diigmg6bqxpixcj3dzzqavmc6783pjs1-reference-graph-legacy.drv'...
/nix/store/1q9vc0lq7qjlfjz47mfmlzdf86c543jy-xgcc-13.2.0-libgcc
0
/nix/store/ic63ay0py10fyryaw7345k4ps32da33w-libidn2-2.3.7
2
/nix/store/ic63ay0py10fyryaw7345k4ps32da33w-libidn2-2.3.7
/nix/store/yfp7dr8m7zi7kxk49wd714gwvhb105hf-libunistring-1.1
/nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52
3
/nix/store/1q9vc0lq7qjlfjz47mfmlzdf86c543jy-xgcc-13.2.0-libgcc
/nix/store/ic63ay0py10fyryaw7345k4ps32da33w-libidn2-2.3.7
/nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52
/nix/store/yfp7dr8m7zi7kxk49wd714gwvhb105hf-libunistring-1.1
1
/nix/store/yfp7dr8m7zi7kxk49wd714gwvhb105hf-libunistring-1.1
/nix/store/yh6x9ia5kxxw3w90vkb09vqgfvhb416k-hello-2.12.1
2
/nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52
/nix/store/yh6x9ia5kxxw3w90vkb09vqgfvhb416k-hello-2.12.1
/nix/store/ik4m89xg09jx21fkjh54varbj450m5dp-reference-graph-legacy
getExe
, successful buildix-build --no-out-link -E "let lib = import ./lib; pkgs = import ./. { }; path = lib.getExe pkgs.hello; in pkgs.runCommandLocal ''reference-graph-legacy'' { exportReferencesGraph = [ ''graph'' path ]; } ''cat ./graph | tee \$out''"
this derivation will be built:
/nix/store/rlzp3ilxh7xghsn22vnjblccijhg7c29-reference-graph-legacy.drv
building '/nix/store/rlzp3ilxh7xghsn22vnjblccijhg7c29-reference-graph-legacy.drv'...
/nix/store/1q9vc0lq7qjlfjz47mfmlzdf86c543jy-xgcc-13.2.0-libgcc
0
/nix/store/ic63ay0py10fyryaw7345k4ps32da33w-libidn2-2.3.7
2
/nix/store/ic63ay0py10fyryaw7345k4ps32da33w-libidn2-2.3.7
/nix/store/yfp7dr8m7zi7kxk49wd714gwvhb105hf-libunistring-1.1
/nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52
3
/nix/store/1q9vc0lq7qjlfjz47mfmlzdf86c543jy-xgcc-13.2.0-libgcc
/nix/store/ic63ay0py10fyryaw7345k4ps32da33w-libidn2-2.3.7
/nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52
/nix/store/yfp7dr8m7zi7kxk49wd714gwvhb105hf-libunistring-1.1
1
/nix/store/yfp7dr8m7zi7kxk49wd714gwvhb105hf-libunistring-1.1
/nix/store/yh6x9ia5kxxw3w90vkb09vqgfvhb416k-hello-2.12.1
2
/nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52
/nix/store/yh6x9ia5kxxw3w90vkb09vqgfvhb416k-hello-2.12.1
/nix/store/h9bigga3j2ah1dpakjz1bpsyzy6iqvf2-reference-graph-legacy
outPath
, successful buildnix-build --no-out-link -E "let lib = import ./lib; pkgs = import ./. { }; path = pkgs.hello.outPath; in pkgs.runCommandLocal ''reference-graph-structured'' { __structuredAttrs = true; nativeBuildInputs = [ pkgs.buildPackages.jq ]; exportReferencesGraph.graph = [ path ]; } ''jq . \$NIX_ATTRS_JSON_FILE | tee \$out''"
this derivation will be built:
/nix/store/6wrl680daaffrm8hy4ahlkslv7dlq28q-reference-graph-structured.drv
building '/nix/store/6wrl680daaffrm8hy4ahlkslv7dlq28q-reference-graph-structured.drv'...
structuredAttrs is enabled
{
"allowSubstitutes": false,
"buildCommand": "jq . $NIX_ATTRS_JSON_FILE | tee $out",
"buildInputs": [],
"builder": "/nix/store/306znyj77fv49kwnkpxmb0j2znqpa8bj-bash-5.2p26/bin/bash",
"cmakeFlags": [],
"configureFlags": [],
"depsBuildBuild": [],
"depsBuildBuildPropagated": [],
"depsBuildTarget": [],
"depsBuildTargetPropagated": [],
"depsHostHost": [],
"depsHostHostPropagated": [],
"depsTargetTarget": [],
"depsTargetTargetPropagated": [],
"doCheck": false,
"doInstallCheck": false,
"enableParallelBuilding": true,
"enableParallelChecking": true,
"enableParallelInstalling": true,
"env": {},
"exportReferencesGraph": {
"graph": [
"/nix/store/yh6x9ia5kxxw3w90vkb09vqgfvhb416k-hello-2.12.1"
]
},
"graph": [
{
"closureSize": 159560,
"narHash": "sha256:10q8iyvfmpfck3yiisnj1j8vp6lq3km17r26sr95zpdf9mgmk69s",
"narSize": 159560,
"path": "/nix/store/1q9vc0lq7qjlfjz47mfmlzdf86c543jy-xgcc-13.2.0-libgcc",
"references": [],
"valid": true
},
{
"closureSize": 2190120,
"narHash": "sha256:0mfvcix1h7prq9wwpwfbwyj56pkdm3w05i69zwvd911fg27w7ccy",
"narSize": 361136,
"path": "/nix/store/ic63ay0py10fyryaw7345k4ps32da33w-libidn2-2.3.7",
"references": [
"/nix/store/ic63ay0py10fyryaw7345k4ps32da33w-libidn2-2.3.7",
"/nix/store/yfp7dr8m7zi7kxk49wd714gwvhb105hf-libunistring-1.1"
],
"valid": true
},
{
"closureSize": 32473824,
"narHash": "sha256:1ljsk162rvyhjdsfbiiq26lsm8ahz68y1g366r6wwbq31d1aanah",
"narSize": 30124144,
"path": "/nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52",
"references": [
"/nix/store/1q9vc0lq7qjlfjz47mfmlzdf86c543jy-xgcc-13.2.0-libgcc",
"/nix/store/ic63ay0py10fyryaw7345k4ps32da33w-libidn2-2.3.7",
"/nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52"
],
"valid": true
},
{
"closureSize": 1828984,
"narHash": "sha256:1akas0jc72xbc8fz1a8ja1gk5ghh1jqm9s1p46p67sg2bcjfiwzg",
"narSize": 1828984,
"path": "/nix/store/yfp7dr8m7zi7kxk49wd714gwvhb105hf-libunistring-1.1",
"references": [
"/nix/store/yfp7dr8m7zi7kxk49wd714gwvhb105hf-libunistring-1.1"
],
"valid": true
},
{
"closureSize": 32700384,
"narHash": "sha256:1s8ax2azcifcnxsn0ny1gk75hd6hfmm3l5nd56xpsxmya2gwnyg9",
"narSize": 226560,
"path": "/nix/store/yh6x9ia5kxxw3w90vkb09vqgfvhb416k-hello-2.12.1",
"references": [
"/nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52",
"/nix/store/yh6x9ia5kxxw3w90vkb09vqgfvhb416k-hello-2.12.1"
],
"valid": true
}
],
"mesonFlags": [],
"name": "reference-graph-structured",
"nativeBuildInputs": [
"/nix/store/n5chc8i5za5jrr4biwwgx7a96p0a7aq9-jq-1.7.1-dev"
],
"outputs": {
"out": "/nix/store/l30rfsn0lcbnxrd248s40ak22rvrqkm3-reference-graph-structured"
},
"passAsFile": [
"buildCommand"
],
"patches": [],
"preferLocalBuild": true,
"propagatedBuildInputs": [],
"propagatedNativeBuildInputs": [],
"stdenv": "/nix/store/rg6agzlb4gcl4w2126m16qm4mgr8jgi6-stdenv-linux",
"strictDeps": false,
"system": "x86_64-linux"
}
/nix/store/l30rfsn0lcbnxrd248s40ak22rvrqkm3-reference-graph-structured
getExe
, failed buildnix-build --no-out-link -E "let lib = import ./lib; pkgs = import ./. { }; path = lib.getExe pkgs.hello; in pkgs.runCommandLocal ''reference-graph-structured'' { __structuredAttrs = true; nativeBuildInputs = [ pkgs.buildPackages.jq ]; exportReferencesGraph.graph = [ path ]; } ''jq . \$NIX_ATTRS_JSON_FILE | tee \$out''"
this derivation will be built:
/nix/store/n4a18ikia9x5pa5x4hsjvsgb1gnv66wh-reference-graph-structured.drv
error: path '/nix/store/yh6x9ia5kxxw3w90vkb09vqgfvhb416k-hello-2.12.1/bin/hello' is not in the Nix store
The upstream seems to have noticed this. For details, see NixOS/nix#10549 and NixOS/nix#10565.
Ahh great catch! Thanks.
I still think it will be a good idea to have a dedicated "dependencies" or "copyToRoot" field for this, given args is a command array.
Describe the bug
pkgs.ociTools.buildContainer
is failing with errors such aserror: path '/nix/store/306znyj77fv49kwnkpxmb0j2znqpa8bj-bash-5.2p26/bin/bash' is not in the Nix store
.This example in the documentation ( https://nixos.org/manual/nixpkgs/unstable/#ssec-pkgs-ociTools-buildContainer-examples ) fails in this way, as well as when I try to integrate this into my NixOS configuration.
Steps To Reproduce
Follow https://nixos.org/manual/nixpkgs/unstable/#ssec-pkgs-ociTools-buildContainer-examples , e.g.
Expected behavior
I expected an OCI image to be built.
Notify maintainers
@ShamrockLee @aib
Add a :+1: reaction to issues you find important.