NFS / FsEvents failed to run

[qkdreyer]

dinghy up
Starting the dinghy VM...
Starting NFS daemon, this will require sudo
Waiting for NFS daemon...
NFS mounting failed
Starting the FsEvents daemon
Starting DNS and HTTP proxy
Unable to find image 'codekitchen/dinghy-http-proxy:2.5' locally
docker: Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 192.168.64.1:53: read udp 192.168.64.2:53424->192.168.64.1:53: read: connection refused.
See 'docker run --help'.
   VM: running
  NFS: stopped
 FSEV: stopped
  DNS: stopped
PROXY: stopped

NFS failed to run
details available in log file: /usr/local/Cellar/dinghy/4.6.5/local/var/dinghy-NFS.output

FsEvents failed to run
details available in log file: /usr/local/Cellar/dinghy/4.6.5/local/var/dinghy-FsEvents.output

To connect the Docker client to the Docker daemon, please set these environment variables.
You can set them by running:
    eval $(dinghy env)
It's recommended to add this to your shell config such as ~/.bash_profile

    export DOCKER_HOST=tcp://192.168.64.2:2376
    export DOCKER_CERT_PATH=/Users/qkdreyer/.docker/machine/machines/dinghy
    export DOCKER_TLS_VERIFY=1
    export DOCKER_MACHINE_NAME=dinghy

cat /usr/local/Cellar/dinghy/4.6.5/local/var/dinghy-*.output
=== Starting FsEvents at 2019-03-18T11:43:35+01:00 ===

/usr/local/Cellar/dinghy/4.6.5/cli/fsevents_to_vm/lib/fsevents_to_vm/ssh_install_gnu_touch.rb:33:in `install!': Failed to install GNU touch into VM: Unable to find image 'codekitchen/dinghy-http-proxy:2.5' locally (RuntimeError)
Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 192.168.64.1:53: read udp 192.168.64.2:39881->192.168.64.1:53: read: connection refused
    from /usr/local/Cellar/dinghy/4.6.5/cli/fsevents_to_vm/lib/fsevents_to_vm/cli.rb:17:in `start'
    from /usr/local/Cellar/dinghy/4.6.5/cli/thor/lib/thor/command.rb:27:in `run'
    from /usr/local/Cellar/dinghy/4.6.5/cli/thor/lib/thor/invocation.rb:126:in `invoke_command'
    from /usr/local/Cellar/dinghy/4.6.5/cli/thor/lib/thor.rb:359:in `dispatch'
    from /usr/local/Cellar/dinghy/4.6.5/cli/thor/lib/thor/base.rb:440:in `start'
    from /usr/local/Cellar/dinghy/4.6.5/cli/dinghy/fsevents_to_vm.rb:39:in `run'
    from /usr/local/Cellar/dinghy/4.6.5/cli/daemons/lib/daemons/application.rb:270:in `call'
    from /usr/local/Cellar/dinghy/4.6.5/cli/daemons/lib/daemons/application.rb:270:in `block in start_proc'
    from /usr/local/Cellar/dinghy/4.6.5/cli/daemons/lib/daemons/daemonize.rb:84:in `call_as_daemon'
    from /usr/local/Cellar/dinghy/4.6.5/cli/daemons/lib/daemons/application.rb:274:in `start_proc'
    from /usr/local/Cellar/dinghy/4.6.5/cli/daemons/lib/daemons/application.rb:300:in `start'
    from /usr/local/Cellar/dinghy/4.6.5/cli/dinghy/daemon.rb:73:in `start'
    from /usr/local/Cellar/dinghy/4.6.5/cli/dinghy/daemon.rb:14:in `up'
    from /usr/local/Cellar/dinghy/4.6.5/cli/dinghy/fsevents_to_vm.rb:14:in `up'
    from /usr/local/Cellar/dinghy/4.6.5/cli/cli.rb:283:in `start_services'
    from /usr/local/Cellar/dinghy/4.6.5/cli/cli.rb:93:in `up'
    from /usr/local/Cellar/dinghy/4.6.5/cli/thor/lib/thor/command.rb:27:in `run'
    from /usr/local/Cellar/dinghy/4.6.5/cli/thor/lib/thor/invocation.rb:126:in `invoke_command'
    from /usr/local/Cellar/dinghy/4.6.5/cli/thor/lib/thor.rb:359:in `dispatch'
    from /usr/local/Cellar/dinghy/4.6.5/cli/thor/lib/thor/base.rb:440:in `start'
    from /usr/local/bin/_dinghy_command:12:in `<main>'
=== Starting NFS at 2019-03-18T11:43:24+01:00 ===

UNFS3 unfsd 0.9.23 (C) 2009, Pascal Schmidt <unfs3-server@ewetel.net>
bind: Address already in use
Couldn't bind to udp port 19091

docker-machine ls dinghy
NAME     ACTIVE   DRIVER   STATE     URL                       SWARM   DOCKER     ERRORS
dinghy   -        xhyve    Running   tcp://192.168.64.2:2376           v18.09.3

dinghy ssh nslookup google.com
Server:    192.168.64.1
Address 1: 192.168.64.1

nslookup: can't resolve 'google.com'
exit status 1

[codekitchen]

Do you get the same error if you create a machine directly using docker-machine create -d xhyve? In the past this has been an issue with either docker-machine or the xhyve driver, I'm not sure which, and it doesn't seem to affect everybody. It might be worth checking if there's a related issue open with either of those projects.

[qkdreyer]

docker-machine create -d xhyve --xhyve-memory-size 4096 --xhyve-cpu-count 4 --xhyve-disk-size 20000 dinghy
Running pre-create checks...
Creating machine...
(dinghy) Copying /Users/qkdreyer/.docker/machine/cache/boot2docker.iso to /Users/qkdreyer/.docker/machine/machines/dinghy/boot2docker.iso...
(dinghy) Creating VM...
(dinghy) /dev/disk2                                             /Users/qkdreyer/.docker/machine/machines/dinghy/b2d-image
(dinghy) "disk2" ejected.
(dinghy) Generating 20000MB disk image...
(dinghy) created: /Users/qkdreyer/.docker/machine/machines/dinghy/root-volume.sparsebundle
(dinghy) Creating SSH key...
(dinghy) Fix file permission...
(dinghy) Generate UUID...
(dinghy) Convert UUID to MAC address...
(dinghy) Starting dinghy...
(dinghy) Waiting for VM to come online...
(dinghy) Waiting on a pseudo-terminal to be ready... done
(dinghy) Hook up your terminal emulator to /dev/ttys000 in order to connect to your VM
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with boot2docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Checking connection to Docker...
Docker is up and running!
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env dinghy

dinghy ssh nslookup google.com
Server:    192.168.64.1
Address 1: 192.168.64.1

nslookup: can't resolve 'google.com'

dinghy restart
Stopping the FsEvents daemon
Stopping the dinghy VM...
Stopping NFS daemon, this will require sudo
Starting the dinghy VM...
Starting NFS daemon, this will require sudo
Waiting for NFS daemon...
NFS mounting failed
Starting the FsEvents daemon
Starting DNS and HTTP proxy
setting up DNS resolution, this will require sudo
Unable to find image 'codekitchen/dinghy-http-proxy:2.5' locally
docker: Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 192.168.64.1:53: read udp 192.168.64.4:57201->192.168.64.1:53: read: connection refused.
See 'docker run --help'.
   VM: running
  NFS: stopped
 FSEV: stopped
  DNS: stopped
PROXY: stopped

NFS failed to run
details available in log file: /usr/local/Cellar/dinghy/4.6.5/local/var/dinghy-NFS.output

FsEvents failed to run
details available in log file: /usr/local/Cellar/dinghy/4.6.5/local/var/dinghy-FsEvents.output

To connect the Docker client to the Docker daemon, please set these environment variables.
You can set them by running:
    eval $(dinghy env)
It's recommended to add this to your shell config such as ~/.bash_profile

    export DOCKER_HOST=tcp://192.168.64.4:2376
    export DOCKER_CERT_PATH=/Users/qkdreyer/.docker/machine/machines/dinghy
    export DOCKER_TLS_VERIFY=1
    export DOCKER_MACHINE_NAME=dinghy

Doesn't seem to change anything :(

[qkdreyer]

Can I check if setting up DNS resolution, this will require sudo succeded ?

[qkdreyer]

I'm wondering why I'm not seeing https://github.com/codekitchen/dinghy/blob/1a4270ec8cd36ce7e465338943fd3decfe06c1d4/cli/dinghy/http_proxy.rb#L92-L94

dinghy ssh cat /etc/resolv.conf
nameserver 192.168.64.1

[codekitchen]

Darn yeah, this is what I've seen a couple times before. I wish I knew what was causing it, or even whether it was docker-machine vs the xhyve driver.

That resolver file isn't installed on the VM, it is installed on the MacOS host in /etc/resolver/docker, this is why you can reach *.docker domains from your host machine (e.g. Safari) and not just from other Docker containers.

[qkdreyer]

Can't you think about anything you want me to test to find out where that problem is ?

[codekitchen]

I'm not in a great position to figure this one out, I'm afraid. It might be useful to test with Virtualbox, see if you get the same problem with docker-machine create -d virtualbox. That won't guarantee whether it's a problem with the specific driver vs docker-machine itself, but it could still be a useful data point.

Then I'd file an issue with either the docker-machine-driver-xhyve project or the docker-machine project, or look for an existing issue.

[qkdreyer]

Doing a macOS reboot handled the NFS issues (port 19091 was still used by a dead process, invisible using lsof, shown using netstat)

I was still having issues with DNS, which I solved by using 192.16.8.1.1 instead of 192.168.64.1 as resolver.

dinghy ssh nslookup docker.io                                                                                             ✔  10013  16:34:06
Server:    192.168.64.1
Address 1: 192.168.64.1

nslookup: can't resolve 'docker.io'
exit status 1

dinghy ssh "echo 'nameserver 192.168.1.1' | sudo tee /etc/resolv.conf"

dinghy ssh nslookup docker.io                                                                                             ✔  10016  16:34:30
Server:    192.168.1.1
Address 1: 192.168.1.1

Name:      docker.io
Address 1: 3.91.211.1 ec2-3-91-211-1.compute-1.amazonaws.com
Address 2: 52.54.178.62 ec2-52-54-178-62.compute-1.amazonaws.com
Address 3: 52.207.42.240 ec2-52-207-42-240.compute-1.amazonaws.com

[qkdreyer]

I've just found out what the real DNS problem was. I had already a local dnsmasq installed, which was not listening to docker interfaces.

After running these commands, no more DNS issues inside dinghy :

echo 'listen-address=127.0.0.1' >> /usr/local/etc/dnsmasq.conf
echo 'listen-address=192.168.64.1' >> /usr/local/etc/dnsmasq.conf
sudo brew services restart dnsmasq