dockerd "Advanced" custom config is broken by UI

ovanes commented 7 years ago

Expected behavior

dockerd starts properly
Advanced config is always editable, especially after it was applied and docker can't start.

Actual behavior

Once "Advanced" config was applied and dockerd can't start it is disabled and one can't do anything about it in the UI, but investigate where is the actual data is stored and how to fix OR to reset docker to factory setting, which ... (put whatever curse word here)!
dockerd is not started due to escaped daemon.json:

Information

Why are slashes escaped in /etc/docker/daemon.json:

{"hosts":["tcp:\/\/0.0.0.0:2375","unix:\/\/\/var\/run\/docker.sock"]}

Removing them from JSON, makes dockerd start again.

Steps to reproduce the behavior

Attaching to Docker TTY via screen

screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty

and tracing dockerd startup produces:

Freeing unused kernel memory: 1760K (ffff880001848000 - ffff880001a00000)
Freeing unused kernel memory: 1248K (ffff880001cc8000 - ffff880001e00000)

   OpenRC 0.21.7.e3f10ac is starting up Linux 4.9.3-moby (x86_64)

 * Mounting /proc ... [ ok ]
 * Mounting /run ... * /run/openrc: creating directory
 * /run/lock: creating directory
 * /run/lock: correcting owner
 * Caching service dependencies ... [ ok ]
 * Mounting /sys ... [ ok ]
 * Mounting security filesystem ... [ ok ]
 * Mounting debug filesystem ... [ ok ]
 * Mounting fuse control filesystem ... [ ok ]
 * Mounting persistent storage (pstore) filesystem ... [ ok ]
 * Mounting cgroup filesystem ... [ ok ]
 * Mounting devtmpfs on /dev ... [ ok ]
 * Mounting /dev/mqueue ... [ ok ]
 * Mounting /dev/pts ... [ ok ]
 * Mounting /dev/shm ...tsc: Refined TSC clocksource calibration: 2691.922 MHz
clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x26cd6f7a88b, max_idle_ns: 440795245081 ns
 [ ok ]
 * Starting busybox mdev ... [ ok ]
 * Configuring host block device ...boot2docker-data: clean, 706892/1218224 files, 2798912/4871711 blocks
 [ ok ]
 * Loading hardware drivers ...modprobe: module fbcon not found in modules.dep
 [ ok ]
 * Remounting filesystems ... [ ok ]
 * Mounting local filesystems ... [ ok ]
 * Setting system clock using the hardware clock [UTC] ... [ ok ]
 * Setting hostname ... [ ok ]
 * Creating user login records ... [ ok ]
 * Starting vsock proxy ... [ ok ]
 * sysklogd -> start: syslogd ... [ ok ]
 * sysklogd -> start: klogd ... [ ok ]
 * Starting busybox crond ... [ ok ]
 * Mounting misc binary format filesystem ... [ ok ]
 * Setting sysfs variables ... [ ok ]
 * Starting local ... [ ok ]
 * Configuring kernel parameters ... [ ok ]
 * Starting DHCP Client Daemon ... [ ok ]
 * Starting networking ... *   lo ... [ ok ]
 * Initializing random number generator ... [ ok ]
 * Starting busybox acpid ... [ ok ]
 * Starting FUSE socket passthrough ... [ ok ]
 * Running system containerd ... [ ok ]
 * Running system containers ... binfmt rng-tools [ ok ]
 * Configuring host settings from database ... [ ok ]
 * Setting up proxy port service ... [ ok ]
 * Starting Docker ...* Failed to start docker
 [ !! ]
 * ERROR: docker failed to start
 * ERROR: cannot start aws as docker would not start
 * ERROR: cannot start azure as docker would not start
 * Starting chronyd ... [ ok ]
 * Checking system state ...
✓ Drive found: sda
✓ Drive mounted: /dev/sda1 on /var type ext4 (rw,relatime,data=ordered)
✓ Network connected:           inet addr:192.168.65.2  Bcast:192.168.65.7  Mask:255.255.255.248
✓ Process transfused running
✗ No dockerd process
✗ No containerd process
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
✗ Docker ps failed: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
✓ Diagnostics server running: /usr/bin/diagnostics-server -vsock
✓ System containerd server running: /usr/bin/containerd
✓ System containerd working
 * ERROR: diagnostics failed to start
 * Starting Hyper-V daemon: hv_kvp_daemon ... [ ok ]
 * Starting Hyper-V daemon: hv_vss_daemon ... [ ok ]
 * Adjusting oom killer settings ... [ ok ]
 * ERROR: cannot start test as docker would not start

Welcome to Moby

                        ##         .
                  ## ## ##        ==

Removing \ from /etc/docker/daemon.json escape slashes, make dockerd start again:

/ # dockerd
WARN[0000] [!] DON'T BIND ON ANY IP ADDRESS WITHOUT setting -tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING [!]
INFO[0000] libcontainerd: new containerd process, pid: 2077
WARN[0000] containerd: low RLIMIT_NOFILE changing to max  current=1024 max=4096
INFO[0001] [graphdriver] using prior storage driver: aufs
INFO[0002] Graph migration to content-addressability took 0.00 seconds
INFO[0002] Loading containers: start.
WARN[0002] Running modprobe nf_nat failed with message: `modprobe: module nf_nat not found in modules.dep`, error: exit status 1
WARN[0002] Running modprobe xt_conntrack failed with message: `modprobe: module xt_conntrack not found in modules.dep`, error: exit status 1
INFO[0002] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address
INFO[0003] Loading containers: done.
INFO[0003] Daemon has completed initialization
INFO[0003] Docker daemon                                 commit=48a9e53 graphdriver=aufs version=1.13.0-rc7
INFO[0003] API listen on /var/run/docker.sock
INFO[0003] API listen on [::]:2375

Please either fix it asap, or at least reply, where to edit them persistently...

dsheets commented 7 years ago

I am investigating the issue. It looks like there are at least two defects. In the mean time, you can edit the daemon.json manually like so:

$ cd ~/Library/Containers/com.docker.docker/Data/database
$ git reset --hard
$ $EDITOR com.docker.driver.amd64-linux/etc/docker/daemon.json
$ git add com.docker.driver.amd64-linux/etc/docker/daemon.json
$ git commit -m "worked around daemon.json brokenness"

It looks like adding hosts to daemon.json will always fail as Docker for Mac passes hosts as a daemon command line flag as well:

Jan 18 17:14:16 moby root: unable to configure the Docker daemon with file /etc/docker/daemon.json: the following directives are specified both as a flag and in the configuration file: hosts: (from flag: [unix:///var/run/docker.sock], from file: [tcp://0.0.0.0:2375 unix:///var/run/docker.sock])

dsheets commented 7 years ago

Thanks for your report! I've now filed 3 separate bugs (for lock-out, unnecessary escaping, and allowing hosts field). We'll report back here on progress.

ovanes commented 7 years ago

Thanks for reacting so fast, as it is a major breaking point for me right now, to debug our distributed application. I need to connect to docker instances with remote debuggers.

dsheets commented 7 years ago

I recommend using a tool like socat to connect over TCP:

$ socat TCP-LISTEN:2376,reuseaddr,fork,bind=127.0.0.1 UNIX-CLIENT:/var/run/docker.sock

In this case, I've restricted the socket to only listen on the loopback interface but you may want it to be more widely available (or restrict the network with range=CIDR or similar).

rawjeev commented 7 years ago

I couldn't find /etc/docker/daemon.json not even the folder /etc/docker/ on my mac.

Should this file be created manually under the location (the comment above does not suggest that though)

ovanes commented 7 years ago

@rawjeev Did you attach to docker TTY as I described in my post? Or do you just try to find /etc/docker on you Mac OS?

In the section Steps to reproduce the behavior first step I describe is:

Attaching to Docker TTY via screen

screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty

mboogerd commented 7 years ago

@dsheets It seems that the "unnecessary escaping" may be resolved, correct? At least I didn't encounter it when adding hosts to my daemon.conf from the GUI. Unfortunately, I did encounter the other two on version 17.03.1-ce-mac5 (16048), so I wondered whether I could kindly request a status update for at least those two issues? Thanks a bunch in advance!

dsheets commented 7 years ago

The escaping and the lock-out should be fixed. The hosts failure remains.

docker-robott commented 6 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale comment. Stale issues will be closed after an additional 30d of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows. /lifecycle stale

jcrben commented 6 years ago

@dsheets did the hosts failure get fixed?

docker-robott commented 4 years ago

Closed issues are locked after 30 days of inactivity. This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows. /lifecycle locked

docker / for-mac