NixOps blocked waiting for SSH to VirtualBox

[tsenart]

Hello,

While going through the NixOps manual using Mac OSX Yosemite, I ran into the above mentioned issue where SSH access doesn't seem to be available.

$ nix-env --install nixops
...
$ nixops create ./trivial.nix ./trivial-vbox.nix -d trivial
created deployment ‘858b3635-5eed-11e4-ac82-20c9d046b91d’
858b3635-5eed-11e4-ac82-20c9d046b91d
$ nixops list
+--------------------------------------+---------+------------------------+------------+------+
| UUID                                 | Name    | Description            | # Machines | Type |
+--------------------------------------+---------+------------------------+------------+------+
| 858b3635-5eed-11e4-ac82-20c9d046b91d | trivial | Unnamed NixOps network |          0 |      |
+--------------------------------------+---------+------------------------+------------+------+
$ nixops info -d trivial
Network name: trivial
Network UUID: 858b3635-5eed-11e4-ac82-20c9d046b91d
Network description: Web server
Nix expressions: /private/tmp/trivial/trivial.nix /private/tmp/trivial/trivial-vbox.nix

+-----------+---------------+------------+-------------+------------+
| Name      |     Status    | Type       | Resource Id | IP address |
+-----------+---------------+------------+-------------+------------+
| webserver | Missing / New | virtualbox |             |            |
+-----------+---------------+------------+-------------+------------+
$ nixops deploy -d trivial
webserver> creating VirtualBox VM...
webserver> Virtual machine 'nixops-858b3635-5eed-11e4-ac82-20c9d046b91d-webserver' is created and registered.
webserver> UUID: 3c0186a8-c9a5-431b-8cf1-ba4765d4c999
webserver> Settings file: '/Users/tomas/VirtualBox VMs/nixops-858b3635-5eed-11e4-ac82-20c9d046b91d-webserver/nixops-858b3635-5eed-11e4-ac82-20c9d046b91d-webserver.vbox'
webserver> creating disk ‘disk1’...
webserver> 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
webserver> Clone hard disk created in format 'VDI'. UUID: dfa652c2-8c1c-4243-9668-39c47c02f5f8
webserver> attaching disk ‘disk1’...
webserver> Waiting for VM "nixops-858b3635-5eed-11e4-ac82-20c9d046b91d-webserver" to power on...
webserver> VM "nixops-858b3635-5eed-11e4-ac82-20c9d046b91d-webserver" has been successfully started.
webserver> waiting for IP address........................ 192.168.56.101
webserver> waiting for SSH.............................................................................................................

[rened]

I had the same issue and am not entirely sure anymore what fixed it. In the end I used https://github.com/zefhemel/nixops-mac-setup/pull/4 to install it nix/nixops. It uses nix 1.8pre / nixops 1.3pre on the host, as well as updating nix to 1.8pre inside the vbox. Then deployments work just fine.

[tsenart]

Upgrading to nix 1.8pre fixed the issue.

[tsenart]

OK. This broke further down again with the following:

building path(s) ‘/nix/store/8nzwcj432hqxh73q3w62kmzmp5fanljc-users-groups.json’
error: a ‘x86_64-linux’ is required to build ‘/nix/store/h7s2kgq77gw3bv8pr8pac87kvr6r5ab0-users-groups.json.drv’, but I am a ‘x86_64-darwin’

My current nix version is nix-env (Nix) 1.8pre3866_6062b12.

[rbvermaa]

The last issue looks like a nix issue.

[rened]

Again I am not 100% sure, but I think I solved it by installing the last nixops binary. Excerpt from https://github.com/rened/nixops-mac-setup/blob/master/install-nixops.sh:

# get the .nixpkg URL of the latest darwin build of nixops
URL=`curl -sL http://hydra.nixos.org/job/nixops/master/build.x86_64-darwin/latest | grep .nixpkg | grep href | grep -o \".*\" | sed s/\"//g`
nix-install-package --non-interactive --url $URL

try deploying again. Should it then fail with some message related to EOL something, you need to update nix within the VM and redeploy. https://github.com/rened/nixops-mac-setup/blob/master/README.md#use has details on this. The last part (updating nix within the VM) is related to https://github.com/NixOS/nix/issues/243

[tsenart]

@rened: I followed the instructions in your repo from scratch. Here's where things halt:

building path(s) ‘/nix/store/bz5bngwd1sxwxcc1cqdhzgmfsx3z4xd6-stage-2-init.sh’
error: a ‘x86_64-linux’ is required to build ‘/nix/store/3z3z7vy0kgayc2m0vd1fjwwrq9m42xps-stage-2-init.sh.drv’, but I am a ‘x86_64-darwin’
error: unable to build all machine configurations

[tsenart]

Ooops. Replied too fast. I see this mentioned in the README. Uffff... I'll go through it. :-)

[tsenart]

Worked! I suppose I shouldn't have to work through all of this though, so, keeping this issue open.

[rened]

yes, this should be easier ;-)

@rbvermaa do you know what the roadmap for releasing nix 1.8 and nixops 1.3 is? and for virtualbox images with nix 1.8 / nixos 14.11? i am asking because I am planning to work on a update-vm-to-nix-1.8-if-necessary work-around for nix's build-remote.pl - would that make sense? or shall we rather wait for the above versions to be released? thanks!

[ghost]

Just to add a spot of color to this:

When using Nix 1.8pre3903_b0c5c2a and NixOps 1.3pre1362_2a616f8 with nixpkgs from Joel Taylor's Yosemite nixpkgs I get pretty far.

cd nixops/examples
nixops create trivial-vbox.nix trivial.nix -d test
nixops deploy -d test

This gives me the expected (on Darwin) failure to SSH to the virtual machine, because of mismatched Nix versions between Darwin and the virtual machine. Luckily,

nixops ssh machine

works. Once in the shell on the virtual machine:

nix-channel --update
nix-env -iA nixos.pkgs.nixUnstable

Disconnect from the virtual machine, then stop the virtual machine and change its base memory from 512M to something more reasonable, like 2048M, because I ran into problems with what amounted to swap hell before I did this. Make sure to restart the virtual machine afterwards.

Now when I try to deploy again, all the paths are successfully copied to the virtual machine and it starts to build. Essentially the first thing it tries to do is bootstrap GCC 4.8.3. This eventually fails because it can't find a file, crti.o. To determine whether this is an issue bootstrapping GCC 4.8.3 on NixOS generally, I SSH'ed back into the box, and did:

nix-env -iA nixos.pkgs.gcc48 --option use-binary-caches false

This has been running for well over 12 hours at this point :-D because I asked to rebuild everything from source. But it's well past the point at which it failed when building from the paths copied over by nixops.

My tentative conclusion, then, is that there's a bug in nixops 1.3pre1362_2a616f8 such that it incorrectly determines the bootstrap dependencies of at least some packages that differ between platforms, in this case Darwin and NixOS itself. In particular, installing GCC 4.8.3 in NixOS clearly involves something called "bootstrap-tools," which makes sense: you need a C compiler to build GCC. However, I don't think I saw any such dependency pushed over or "built" in the nixops deploy process.

[rbvermaa]

@psnively This looks more like an issue in the nixpkgs branch/clone you are using, not a nixops issue?

[ghost]

OK. How would I go about making that determination and tracking it down? It seems to me that nixops shouldn't care, as long as all of the dependencies specified by the expressions deployed are satisfied. What am I missing?

[domenkozar]

This clearly hints that Virtualbox image didn't come up (well specifically DHCP server in virtualbox didn't assign an IP to the guest). Could we display dmesg from virtualbox or similar?