Tests fail due to failure to download curl source

[obadz]

http://hydra.nixos.org/build/37583672/nixlog/5/tail-reload

machine# building path(s) ‘/nix/store/ri8dxcvazsip0727gxgkw2nnlhgch52q-curl-7.47.1.tar.bz2’
machine# downloading ‘http://curl.haxx.se/download/curl-7.47.1.tar.bz2’... 
machine# error: unable to download ‘http://curl.haxx.se/download/curl-7.47.1.tar.bz2’: Couldn't resolve host name (6)
machine# builder for ‘/nix/store/by5si5whqjhpdvph061d3pkvzz05x4qx-curl-7.47.1.tar.bz2.drv’ failed with exit code 1

Reproduced locally but I can download curl fine. What does this mean?

[dezgeg]

That's happening because it's trying to build 'lndir', which is not included in the CD. I'm guessing that's coming from the symlinkJoin in 1e53d4a7776acbf61f42c094c103652c8068ad64

[obadz]

@ericsagnes / @dezgeg, is there a quick workaround that we could deploy? otherwise we should revert…

[dezgeg]

The quick fix would be:

diff --git a/nixos/tests/installer.nix b/nixos/tests/installer.nix
index 4a30cc1..6cd5592 100644
--- a/nixos/tests/installer.nix
+++ b/nixos/tests/installer.nix
@@ -206,6 +206,7 @@ let
                 pkgs.docbook5
                 pkgs.docbook5_xsl
                 pkgs.unionfs-fuse
+                pkgs.lndir
                 pkgs.ntp
                 pkgs.nixos-artwork
                 pkgs.perlPackages.XMLLibXML

But it might also be possible to change the code to use buildEnv instead of symlinkJoin.

[obadz]

Couldn't make it work with

                pkgs.xorg.lndir
                pkgs.fontconfig
                pkgs.libxml2
                pkgs.libxslt
                pkgs.curl

@ericsagnes: reverting 1e53d4a for now in the hope that the tests will succeed and that we get a release.

We can merge it back in once changes are made such that nix-build '<nixos/release.nix>' -A tests.installer.simple succeeds.

cc @vcunat

[ericsagnes]

@obadz Thanks for reverting! I will make a new PR when I can get everything work fine.

[vcunat]

I'm still getting the problem, though in a different test now (luksroot): https://github.com/NixOS/nixpkgs/commit/ec072cbc4c8#commitcomment-18275251

[obadz]

@abbradar, do you think f4ea97a will fix the luksroot test?

[abbradar]

Yes, it should. Sorry again, the whole thing was because of me doing a bad rebase ^_^;;

I'll try to build the test on my machine to maybe get the results faster.

[obadz]

@abbradar, still doesn't work for me.

I've pushed 08fe395 — shows that the luksroot test is attempting to download libxml2

[abbradar]

Eh, what? I'm unsure why would it need it. Still waiting for all things for the test to download...

For the record: you can try to run nix-build -A tests.installer.luksroot.x86_64-linux nixos/release.nix locally instead of pushing patches to look what Hydra does.

P.S.: here it is running locally, I'll post the results in a sec...

[obadz]

I know how to run it locally :-) but imho it is helpful when glancing at hydra logs to know what it's trying to download.

[abbradar]

I can't get it -- I've tried to add libxml2 and libxslt but now it wants to build a whole bunch of new stuff (including X.org libraries!). I think I'll run a bisect.

And yes, your patch adding curl is useful -- thanks!

[obadz]

bisect is the way but it will be hard because 1e53d4a seemed to have the same effect and it was reverted

[abbradar]

I want to try constructing my own alternative tree where 1e53d4a was never introduced and try bisecting that one.

[obadz]

@abbradar you might end up building the world multiple times

[abbradar]

@obadz I left that to work through the night, hope it would get some results... I also did a manual look through recent commits but have no ideas what can cause this.

I think it would be very nice to have an ability in Nix to build a graph of dependencies, or a sorted tree like this:

-foo
  - libbar
    - glibc
  - libbaz
    - libquz
      - glibc

It would make debugging those issues much simpler. ...wait a minute, I think I can try to script this...

[abbradar]

Even better, we already do have this: nix-store --query --tree store-path (and also --graph). Now we need to get a good and a bad store paths and compare those trees...

[dezgeg]

This is really weird. It's somehow deciding to rebuild the manual due to it (and some other places) referring to nixpkgs version. If I do this it works:

diff --git a/nixos/doc/manual/default.nix b/nixos/doc/manual/default.nix
index 50b1bb9..a393f1d 100644
--- a/nixos/doc/manual/default.nix
+++ b/nixos/doc/manual/default.nix
@@ -45,7 +45,7 @@ let
       exit 1
     fi
     ${libxslt.bin}/bin/xsltproc \
-      --stringparam revision '${revision}' \
+      --stringparam revision 'REV-WAS-HERE' \
       -o $out ${./options-to-docbook.xsl} $optionsXML
   '';

@@ -61,7 +61,7 @@ let
       cp ${../../modules/security/acme.xml} configuration/acme.xml
       cp ${../../modules/i18n/input-method/default.xml} configuration/input-methods.xml
       ln -s ${optionsDocBook} options-db.xml
-      echo "${version}" > version
+      echo "VER-WAS-HERE" > version
     '';

   toc = builtins.toFile "toc.xml"
diff --git a/nixos/modules/tasks/filesystems.nix b/nixos/modules/tasks/filesystems.nix
index cf8232c..4e14d69 100644
--- a/nixos/modules/tasks/filesystems.nix
+++ b/nixos/modules/tasks/filesystems.nix
@@ -44,7 +44,7 @@ let
         default = [ "defaults" ];
         example = [ "data=journal" ];
         description = "Options used to mount the file system.";
-      } // (if versionAtLeast lib.nixpkgsVersion "16.09" then {
+      } // (if true then {
         type = types.listOf types.str;
       } else {
         type = types.either types.commas (types.listOf types.str);

I have no idea how that has anything to do with just the luksroot test, though...

[dezgeg]

Ah, no, all VM installer tests are affected. I thought I was going insane first... So the problem seems to be that when building from a git repo, during VM evaluation time the values are as follows:

trace: revision: 08fe395
trace: version: 16.09.git.08fe395
trace: lib.nixpkgsVersion: 16.09pre-git

But at nixos-install time they are:

machine# trace: lib.nixpkgsVersion: 16.09.git.08fe395
machine# trace: revision: master
machine# trace: version: 16.09.git.08fe395

[abbradar]

Maybe we just want to disable the manual for installer tests? I don't think it's important for the test itself and it's less things to track in extraDependencies...

[obadz]

I don't even understand what the manual has to do with the installer?

[abbradar]

@obadz NixOS builds and installs a manual for itself by default on a system. See services.nixosManual.enable. This option is enabled by default, and if we don't disable it in test configurations for installer tests, the installer attempts to build it for the test installation, too.

P.S.: but I still don't have any idea what has changed now so that the test is broken. Bisect is still not finished ~_~

[obadz]

As of 08fe395, the luksroot installer test is succeeding. Any ideas why?!

[dezgeg]

It is probably dependent on whether it's built from a git checkout or not. (E.g. .git is probably not present in hydra)

[obadz]

Oh. But why would .git be there in 9117d20 but not in 08fe395 ?

[obadz]

Oddly, this does not fix it:

diff --git a/nixos/tests/installer.nix b/nixos/tests/installer.nix
index e71168a..53ac5ac 100644
--- a/nixos/tests/installer.nix
+++ b/nixos/tests/installer.nix
@@ -215,6 +215,7 @@ let
                 # curl's tarball, we see what it's trying to download
                 pkgs.curl
               ]
+              ++ (let manual = import ../../doc; in manual.nativeBuildInputs)
               ++ optional (bootLoader == "grub" && grubVersion == 1) pkgs.grub
               ++ optionals (bootLoader == "grub" && grubVersion == 2) [ pkgs.grub2 pkgs.grub2_efi ];

[obadz]

This "fixes" it:

diff --git a/nixos/tests/installer.nix b/nixos/tests/installer.nix
index e71168a..71fea1f 100644
--- a/nixos/tests/installer.nix
+++ b/nixos/tests/installer.nix
@@ -35,6 +35,7 @@ let
         ''}

         hardware.enableAllFirmware = lib.mkForce false;
+        services.nixosManual.enable = false;

         ${replaceChars ["\n"] ["\n  "] extraConfig}
       }