nix-copy-closure not working

[ephox-services]

I have a nix package that wraps Amazon Dynamodb Local. I wish to build it on a nixos machine, then nix-copy-closure it to another machine (in this case, a Fedora workstation).

As part of this, I wish to "pin" nixpkgs to a specific version for build repeatability. To achieve this, I'm following the approach detailed here: https://gist.github.com/zimbatm/de5350245874361762b6a4dfe5366530

It seems that nix-copy-closure can't deal with this pinning correctly.

My steps are:

Source machine:

• nix-build
• rsync the source
• nix-copy-closure to dest machine

Dest machine:

• nix-env -qaP -f default.nix | grep dynamo

Expected:

• dynamo-db is listed as an available package

Actual:

• an error is shown - see below.

Logs:

Source machine:

/nix/store/f1rkxkk10xqfh3ykpvg93wnpp57x2lf6-dynamodb-local-1.0.0

[ephox@nixnode1:~/jenkins/workspace/nix-dynamodb-local_master-26AJM4ANKOOCRJ26ILBBBRG7JGEQP4T766VACHLVFGLBIL4EKWMQ]$ rsync -aH --delete ./ dylan@nook:/tmp/dyno/
dylan@nook's password: 

[ephox@nixnode1:~/jenkins/workspace/nix-dynamodb-local_master-26AJM4ANKOOCRJ26ILBBBRG7JGEQP4T766VACHLVFGLBIL4EKWMQ]$ nix-copy-closure --to dylan@nook /nix/store/f1rkxkk10xqfh3ykpvg93wnpp57x2lf6-dynamodb-local-1.0.0
dylan@nook's password: 
copying 23 missing paths (150.09 MiB) to ‘dylan@nook’...

Target machine:

[dylan@nook dyno]$ pwd
/tmp/dyno
[dylan@nook dyno]$ ls
[dylan@nook dyno]$ ls /nix/store | grep dynamo
4f3qvlsgaax4c79414yx2n192sf7kz80-amazonka-dynamodb-1.4.5-doc
cnm0fmwa5whm14iri2zyp7k83zb8p145-amazonka-dynamodb-1.4.5.tar.gz.drv
qvdhan654ww3bbyq3cy5d0w7n156v1sy-amazonka-dynamodb-1.4.5
ygvzim6p2jhy1y5n46h3pkkmyrm76171-amazonka-dynamodb-1.4.5.drv
[dylan@nook dyno]$ ls
default.nix  Makefile     nixpin.nix   readme.md  update_nixpin.sh
Jenkinsfile  nixpin.json  package.nix  result
[dylan@nook dyno]$ rm result
[dylan@nook dyno]$ ls
default.nix  Makefile     nixpin.nix   readme.md
Jenkinsfile  nixpin.json  package.nix  update_nixpin.sh
[dylan@nook dyno]$ nix-env -qaP nix-dynamodb-local
error: selector ‘nix-dynamodb-local’ matches no derivations
[dylan@nook dyno]$ nix-env -qaP | grep dynamo
[dylan@nook dyno]$ ls /nix/store | grep dynamo
4f3qvlsgaax4c79414yx2n192sf7kz80-amazonka-dynamodb-1.4.5-doc
cnm0fmwa5whm14iri2zyp7k83zb8p145-amazonka-dynamodb-1.4.5.tar.gz.drv
f1rkxkk10xqfh3ykpvg93wnpp57x2lf6-dynamodb-local-1.0.0
qvdhan654ww3bbyq3cy5d0w7n156v1sy-amazonka-dynamodb-1.4.5
ygvzim6p2jhy1y5n46h3pkkmyrm76171-amazonka-dynamodb-1.4.5.drv
[dylan@nook dyno]$ nix-env -qaP -f default.nix 
error: cannot import ‘/nix/store/v2w72ydhc2sygmaqnzhfw7b04z3i54mv-nixpkgs-channels-25f4906da6387e132823417bc54ea86040fb9bd5-src’, since path ‘/nix/store/pkb1w9v39kz26fm109si5d6hxb699q32-nixpkgs-channels-25f4906da6387e132823417bc54ea86040fb9bd5-src.drv’ is not valid, at /tmp/dyno/nixpin.nix:5:3
(use ‘--show-trace’ to show detailed location information)
[dylan@nook dyno]$ nix-env -qaP -f default.nix --show-trace
error: while evaluating anonymous function at /tmp/dyno/default.nix:1:1, called from undefined position:
while evaluating anonymous function at /tmp/dyno/package.nix:1:1, called from /tmp/dyno/default.nix:3:1:
while evaluating the file ‘/tmp/dyno/nixpin.nix’:
cannot import ‘/nix/store/v2w72ydhc2sygmaqnzhfw7b04z3i54mv-nixpkgs-channels-25f4906da6387e132823417bc54ea86040fb9bd5-src’, since path ‘/nix/store/pkb1w9v39kz26fm109si5d6hxb699q32-nixpkgs-channels-25f4906da6387e132823417bc54ea86040fb9bd5-src.drv’ is not valid, at /tmp/dyno/nixpin.nix:5:3

Code is here: https://github.com/ephox-services/nix-copy-closure-bug-replication

Thanks!

[ephox-services]

Note: I've also experienced this error when using channels. I constructed a channel out of these sources and got this error when the client was attempting to update the channel.

[puffnfresh]

nix-copy-closure doesn't do anything with nix-env. You can add it to nix-env afterwards by doing a nix-env -i on the derivation.

[dezgeg]

Indeed, the usage of copy-closure isn't related to the error you're getting. I've minimized the testcase to:

let
  _nixpkgs = import <nixpkgs> {};
  nixpkgsSrc = _nixpkgs.fetchFromGitHub {
    owner = "NixOS";
    repo = "nixpkgs-channels";
    rev = "25f4906da6387e132823417bc54ea86040fb9bd5";
    sha256 = "0pa01hfsz1ddma5d3x41i049wrn176sggr290wpbbhw9arx3nx2i";
  };
  nixpkgs = import nixpkgsSrc {};
in {
  inherit (nixpkgs) hello;
}

which still gives the same on nix-env -f repro.nix -qaP:

error: cannot import ‘/nix/store/v2w72ydhc2sygmaqnzhfw7b04z3i54mv-nixpkgs-channels-25f4906da6387e132823417bc54ea86040fb9bd5-src’, since path ‘/nix/store/pkb1w9v39kz26fm109si5d6hxb699q32-nixpkgs-channels-25f4906da6387e132823417bc54ea86040fb9bd5-src.drv’ is not valid, at /tmp/nix-copy-closure-bug-replication/repro.nix

So it's somehow related to import-from-derivation inside nix-env -q.

[puffnfresh]

@dezgeg the dynamic nixpkgs import is probably the problem. Using fetchTarball will probably fix this.

See https://github.com/NixOS/nixpkgs/issues/27994 for more discussion about nixpkgs pinning.

[benkolera]

@puffnfresh : fetchTarball does indeed resolve the issue in the repro case. for reference: the tarball url is:

https://github.com/NixOS/nixpkgs-channels/archive/96457d26dded05bcba8e9fbb9bf0255596654aab.tar.gz

[benkolera]

To be clear, to get it working I changed repro to this. This sidesteps the need for a bootstrapping nixpkgs and just uses the builtin. Is this what you meant? Obviously in real code we'll need the new fetchTarball that can take a sha. :)

let
  nixpkgsSrc = fetchTarball { url = https://github.com/NixOS/nixpkgs-channels/archive/96457d26dded05bcba8e9fbb9bf0255596654aab.tar.gz; };
  nixpkgs = import nixpkgsSrc {};
in {
  inherit (nixpkgs) hello;
}

[globin]

nix-env runs nix in read-only mode => IFD does not work there

[benkolera]

Forgive my ignorance, but what is IFD, @globin ?

[puffnfresh]

@benkolera "import from derivation" - meaning you use Nix to build an output then you import from that output. The fetchFromGitHub is a Nix derivation, so you're importing the output of that.

Instead, fetchTarball is a special builtin which isn't a derivation. It's some magic instead :ghost:

[stale[bot]]

I marked this as stale due to inactivity. → More info