Deploy times out trying to SSH to VirtualBox VM

exarkun commented 5 years ago

Given:

{
  machine =
    { config, pkgs, ... }:
    { deployment.targetEnv = "virtualbox";
      deployment.virtualbox.memorySize = 8192; # megabytes
      deployment.virtualbox.vcpu = 1; # number of cpus
      deployment.virtualbox.headless = true;
    };
}

This behavior results:

$ nixops deploy -d vbox
machine> creating VirtualBox VM...
machine> Virtual machine 'nixops-0e66b75e-bb66-11e8-96a8-0242c34cc268-machine' is created and registered.
machine> UUID: c33a667a-634c-4e58-bd19-19ea57c9018f
machine> Settings file: '/home/exarkun/VirtualBox VMs/nixops-0e66b75e-bb66-11e8-96a8-0242c34cc268-machine/nixops-0e66b75e-bb66-11e8-96a8-0242c34cc268-machine.vbox'
machine> creating disk ‘disk1’...
machine> 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
machine> Clone medium created in format 'VDI'. UUID: ccda1712-4908-48da-8a74-83d982ab74be
machine> attaching disk ‘disk1’...
machine> Waiting for VM "nixops-0e66b75e-bb66-11e8-96a8-0242c34cc268-machine" to power on...
machine> VM "nixops-0e66b75e-bb66-11e8-96a8-0242c34cc268-machine" has been successfully started.
machine> waiting for IP address........................ 192.168.56.101
ssh: connect to host 192.168.56.101 port 22: Connection timed out
machine> could not connect to ‘root@192.168.56.101’, retrying in 1 seconds...
ssh: connect to host 192.168.56.101 port 22: Connection timed out
machine> could not connect to ‘root@192.168.56.101’, retrying in 2 seconds...
...

$ vboxmanage --version
5.2.10_Ubuntur121806
$ nixops --version
NixOps 1.6
$ vboxmanage list dhcpservers
NetworkName:    HostInterfaceNetworking-vboxnet0
IP:             192.168.56.100
NetworkMask:    255.255.255.0
lowerIPAddress: 192.168.56.101
upperIPAddress: 192.168.56.254
Enabled:        Yes

$ vboxmanage list runningvms
"nixops-0e66b75e-bb66-11e8-96a8-0242c34cc268-machine" {c33a667a-634c-4e58-bd19-19ea57c9018f}

rawtaz commented 5 years ago

Sorry to say that I have yet to use NixOps, but some silly ideas to check/verify:

Is there a firewall of any kind in the VM? If yes, try adding e.g. networking.firewall.allowedTCPPorts = [ 22 ];
Is the SSH service running? Perhaps try adding e.g. services.ssh.enable = true; or services.openssh.enable = true;, whichever applies.
At the same time that NixOps is trying to SSH to the VM, see if you can ping the VM on the IP that is presented.
You mentioned out of band that the created VM was configured with NAT as the network protocol. If I were in your shoes I'd see if it's possible to configure port forwarding when the VM is in NAT mode - if it is, then perhaps that might be something NixOps could be made to support in the future.

I'm not saying any of these actions should be necessary, but for debugging purposes they're of interest.

exarkun commented 5 years ago

Is there a firewall of any kind in the VM?

I don't know if there is. I can't access it. :) I don't know where the definition of the VirtualBox image is.

Is the SSH service running?

Also don't know .

At the same time that NixOps is trying to SSH to the VM, see if you can ping the VM on the IP that is presented.

The IP address is apparently a black hole. Pings are not returned.

rawtaz commented 5 years ago

Is the VM deleted? If not, you should be able to open up VirtualBox, start the VM in non-headless mode, and access it through the regular terminal (unless there's something different with these VMs).

exarkun commented 5 years ago

I can start it in non-headless but I don't know the login details.

rawtaz commented 5 years ago

You seem so uninterested in trying to figure it out, so I'll refrain from trying to come up with ideas. Good luck.

LucianU commented 4 years ago

I have this issue as well. As far as I can tell, nixops is creating the VM, so all the answers to the questions above should be with the people who work on nixops.

teto commented 4 years ago

if you wait ~30mn, does it work ? I have a similar issue https://github.com/NixOS/nixops/issues/1199 so it might not be virtualbox specific.

sbrow commented 2 years ago

Disappointing to see that this still seems to be an issue.

I am runing Virtualbox 6.1.30 on MacOS Monterey (12.4) Intel.

I have a nixos docker image being used as a remote builder.

The virtualbox image gets created fine, but I can't log in to it. Ssh is running (confirmed with nmap -Pn -p 22), The keys get generated, but even when I copy it the key to the mac fs, I still cant log in.

I notice that inside of /nix/store/clysdxwghn0yi9r7yignhjh299xlpsa9-nixops-1.7/share/nix/nixops/virtualbox-image-nixops.nix There is no mention of services.openssh.permitRootLogin = "yes", perhaps this is the issue?

I tried nixops v2, but I couldn't even get it to build. :(

rawtaz commented 2 years ago

@sbrow What command do you use when trying to SSH to the VM? What does that command along with -vvv output? It would be interesting to see what ssh thinks and whether it even tries to use the keys.

sbrow commented 2 years ago

@rawtaz Note: I only included -F none here to hide my config files from this output.

$ ssh -F none -vvv -i ~/.ssh/nixops_rsa root@169.254.68.233                                                                                                                    spencer@ai-geneator
OpenSSH_8.6p1, LibreSSL 3.3.6
debug2: resolve_canonicalize: hostname 169.254.68.233 is address
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/Users/spencer/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/Users/spencer/.ssh/known_hosts2'
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug3: ssh_connect_direct: entering
debug1: Connecting to 169.254.68.233 [169.254.68.233] port 22.
debug3: set_sock_tos: set socket 3 IP_TOS 0x48
debug1: connect to address 169.254.68.233 port 22: Operation timed out
ssh: connect to host 169.254.68.233 port 22: Operation timed out

and the nmap:

 nmap -p 22 -Pn 169.254.68.233                                                                                                                                               spencer@ai-geneator
Starting Nmap 7.92 ( https://nmap.org ) at 2022-06-28 11:49 EDT
Nmap scan report for 169.254.68.233
Host is up.

PORT   STATE    SERVICE
22/tcp filtered ssh

Nmap done: 1 IP address (1 host up) scanned in 2.16 seconds

I got ~/.ssh/nixops_rsa from mounting the disk1.vdi to another VM and copying the file /etc/ssh/ssh_host_ed25519_key

I am running a Pi-hole on my network as DNS and DHCP, but I don't think that's causing my issue.

sbrow commented 2 years ago

Leaving this here for anyone that has the same problems.

Minimal nix config:

let
  hostname = "127.0.0.1";
in
{
  network.description = "Virtualbox machine";

  sample =
    { config, pkgs, ... }:
    {
      deployment.targetEnv = "virtualbox";
      deployment.virtualbox = {
        headless = true;
        vcpu = 1;
        memorySize = 2048;
        vmFlags = [
          "--natpf1" "ssh,tcp,${hostname},3322,,22"
        ];
      };
      deployment.targetPort = 3322;
      # deployment.targetHost = hostname;

      services.openssh.enable = true;
      services.openssh.permitRootLogin = "yes";
    };
}

And in your ~/.ssh/config:

Host 169.254.72.5 # Replace with the IP output from 'nixops deploy'
    User root
    HostName 127.0.0.1
    Port 3322

NixOS / nixops

Deploy times out trying to SSH to VirtualBox VM #1009