Overlay Network : Unable ping container from one host to another

[sujeetkp]

I am trying to create an overlay network without swarm.

I have 3 hosts. (2 ubuntu and 1 centos)

On one of the ubuntu hosts (ubuntu host 1) I am running the consul container.

docker run -d -p 8500:8500 -h consul --name consul progrium/consul -server -bootstrap

After that I have started the docker daemons on the other ubuntu host (ubuntu host 2) and centos host with the below command

sudo /usr/bin/dockerd -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock --cluster-advertise docker0:2375 --cluster-store consul://<public-ip-ubuntu host 1>:8500

Then I created the overlay network as below.

docker network create -d overlay --subnet=192.168.3.0/24 my-overlay

I can see my overlay network on both the hosts.

After that I created containerA on ubuntu host 2 and containerB on centos.

docker run -itd --name containerA --net my-overlay busybox

docker run -itd --name containerB --net my-overlay busybox

But I am not able to ping one container from the other.

root@instance-1:~# docker exec containerA ping -w 5 containerB PING containerB (192.168.3.3): 56 data bytes

--- containerB ping statistics --- 5 packets transmitted, 0 packets received, 100% packet loss root@instance-1:~#

[root@docker-2 ~]# docker exec containerB ping -w 5 containerA PING containerA (192.168.3.2): 56 data bytes

--- containerA ping statistics --- 5 packets transmitted, 0 packets received, 100% packet loss You have new mail in /var/spool/mail/root

root@instance-1:~# docker inspect containerA [ { "Id": "c34854d53ce7613c3a20f73f4d429b4ec1e3a6a3ba0cc50c257bb471f5184e7e", "Created": "2017-12-28T17:42:37.47345131Z", "Path": "sh", "Args": [], "State": { "Status": "running", "Running": true, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 3871, "ExitCode": 0, "Error": "", "StartedAt": "2017-12-28T17:57:29.578002102Z", "FinishedAt": "2017-12-28T17:57:06.102289772Z" }, "Image": "sha256:6ad733544a6317992a6fac4eb19fe1df577d4dec7529efec28a5bd0edad0fd30", "ResolvConfPath": "/var/lib/docker/containers/c34854d53ce7613c3a20f73f4d429b4ec1e3a6a3ba0cc50c257bb471f5184e7e/resolv.conf", "HostnamePath": "/var/lib/docker/containers/c34854d53ce7613c3a20f73f4d429b4ec1e3a6a3ba0cc50c257bb471f5184e7e/hostname", "HostsPath": "/var/lib/docker/containers/c34854d53ce7613c3a20f73f4d429b4ec1e3a6a3ba0cc50c257bb471f5184e7e/hosts", "LogPath": "/var/lib/docker/containers/c34854d53ce7613c3a20f73f4d429b4ec1e3a6a3ba0cc50c257bb471f5184e7e/c34854d53ce7613c3a20f73f4d429b4ec1e3a6a3ba0cc50c257bb471f5184e7e-json.log", "Name": "/containerA", "RestartCount": 0, "Driver": "aufs", "MountLabel": "", "ProcessLabel": "", "AppArmorProfile": "docker-default", "ExecIDs": null, "HostConfig": { "Binds": null, "ContainerIDFile": "", "LogConfig": { "Type": "json-file", "Config": {} }, "NetworkMode": "my-overlay", "PortBindings": {}, "RestartPolicy": { "Name": "no", "MaximumRetryCount": 0 }, "AutoRemove": false, "VolumeDriver": "", "VolumesFrom": null, "CapAdd": null, "CapDrop": null, "Dns": [], "DnsOptions": [], "DnsSearch": [], "ExtraHosts": null, "GroupAdd": null, "IpcMode": "", "Cgroup": "", "Links": null, "OomScoreAdj": 0, "PidMode": "", "Privileged": false, "PublishAllPorts": false, "ReadonlyRootfs": false, "SecurityOpt": null, "UTSMode": "", "UsernsMode": "", "ShmSize": 67108864, "Runtime": "runc", "ConsoleSize": [ 0, 0 ], "Isolation": "", "CpuShares": 0, "Memory": 0, "NanoCpus": 0, "CgroupParent": "", "BlkioWeight": 0, "BlkioWeightDevice": null, "BlkioDeviceReadBps": null, "BlkioDeviceWriteBps": null, "BlkioDeviceReadIOps": null, "BlkioDeviceWriteIOps": null, "CpuPeriod": 0, "CpuQuota": 0, "CpuRealtimePeriod": 0, "CpuRealtimeRuntime": 0, "CpusetCpus": "", "CpusetMems": "", "Devices": [], "DeviceCgroupRules": null, "DiskQuota": 0, "KernelMemory": 0, "MemoryReservation": 0, "MemorySwap": 0, "MemorySwappiness": -1, "OomKillDisable": false, "PidsLimit": 0, "Ulimits": null, "CpuCount": 0, "CpuPercent": 0, "IOMaximumIOps": 0, "IOMaximumBandwidth": 0 }, "GraphDriver": { "Data": null, "Name": "aufs" }, "Mounts": [], "Config": { "Hostname": "c34854d53ce7", "Domainname": "", "User": "", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "Tty": true, "OpenStdin": true, "StdinOnce": false, "Env": [ "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" ], "Cmd": [ "sh" ], "ArgsEscaped": true, "Image": "busybox", "Volumes": null, "WorkingDir": "", "Entrypoint": null, "OnBuild": null, "Labels": {} }, "NetworkSettings": { "Bridge": "", "SandboxID": "6849fbb88356ea9dae5a3a756b187277de535d29ea2f20d1291dfcdc37a456de", "HairpinMode": false, "LinkLocalIPv6Address": "", "LinkLocalIPv6PrefixLen": 0, "Ports": {}, "SandboxKey": "/var/run/docker/netns/6849fbb88356", "SecondaryIPAddresses": null, "SecondaryIPv6Addresses": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "MacAddress": "", "Networks": { "my-overlay": { "IPAMConfig": null, "Links": null, "Aliases": [ "c34854d53ce7" ], "NetworkID": "cc0db79b09b56e2855799debeb08b17d4c570a63f940fa3bea84212695f040e6", "EndpointID": "b178364726482e9444a1dff3acba64d3b241eeab989647610a4f591a4ddbad28", "Gateway": "", "IPAddress": "192.168.3.2", "IPPrefixLen": 24, "IPv6Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "MacAddress": "02:42:c0:a8:03:02", "DriverOpts": null } } } } ]

[root@docker-2 ~]# docker inspect containerB [ { "Id": "29044717920e764da7a989c4c409ea73fd241174bcefba2ecfd03ecf5b23eb4f", "Created": "2017-12-28T17:33:49.881385212Z", "Path": "sh", "Args": [], "State": { "Status": "running", "Running": true, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 15263, "ExitCode": 0, "Error": "", "StartedAt": "2017-12-28T17:59:22.926608023Z", "FinishedAt": "2017-12-28T17:59:05.940642817Z" }, "Image": "sha256:6ad733544a6317992a6fac4eb19fe1df577d4dec7529efec28a5bd0edad0fd30", "ResolvConfPath": "/var/lib/docker/containers/29044717920e764da7a989c4c409ea73fd241174bcefba2ecfd03ecf5b23eb4f/resolv.conf", "HostnamePath": "/var/lib/docker/containers/29044717920e764da7a989c4c409ea73fd241174bcefba2ecfd03ecf5b23eb4f/hostname", "HostsPath": "/var/lib/docker/containers/29044717920e764da7a989c4c409ea73fd241174bcefba2ecfd03ecf5b23eb4f/hosts", "LogPath": "/var/lib/docker/containers/29044717920e764da7a989c4c409ea73fd241174bcefba2ecfd03ecf5b23eb4f/29044717920e764da7a989c4c409ea73fd241174bcefba2ecfd03ecf5b23eb4f-json.log", "Name": "/containerB", "RestartCount": 0, "Driver": "devicemapper", "MountLabel": "", "ProcessLabel": "", "AppArmorProfile": "", "ExecIDs": null, "HostConfig": { "Binds": null, "ContainerIDFile": "", "LogConfig": { "Type": "json-file", "Config": {} }, "NetworkMode": "my-overlay", "PortBindings": {}, "RestartPolicy": { "Name": "no", "MaximumRetryCount": 0 }, "AutoRemove": false, "VolumeDriver": "", "VolumesFrom": null, "CapAdd": null, "CapDrop": null, "Dns": [], "DnsOptions": [], "DnsSearch": [], "ExtraHosts": null, "GroupAdd": null, "IpcMode": "", "Cgroup": "", "Links": null, "OomScoreAdj": 0, "PidMode": "", "Privileged": false, "PublishAllPorts": false, "ReadonlyRootfs": false, "SecurityOpt": null, "UTSMode": "", "UsernsMode": "", "ShmSize": 67108864, "Runtime": "runc", "ConsoleSize": [ 0, 0 ], "Isolation": "", "CpuShares": 0, "Memory": 0, "NanoCpus": 0, "CgroupParent": "", "BlkioWeight": 0, "BlkioWeightDevice": null, "BlkioDeviceReadBps": null, "BlkioDeviceWriteBps": null, "BlkioDeviceReadIOps": null, "BlkioDeviceWriteIOps": null, "CpuPeriod": 0, "CpuQuota": 0, "CpuRealtimePeriod": 0, "CpuRealtimeRuntime": 0, "CpusetCpus": "", "CpusetMems": "", "Devices": [], "DeviceCgroupRules": null, "DiskQuota": 0, "KernelMemory": 0, "MemoryReservation": 0, "MemorySwap": 0, "MemorySwappiness": -1, "OomKillDisable": false, "PidsLimit": 0, "Ulimits": null, "CpuCount": 0, "CpuPercent": 0, "IOMaximumIOps": 0, "IOMaximumBandwidth": 0 }, "GraphDriver": { "Data": { "DeviceId": "30", "DeviceName": "docker-8:1-25190669-966cdbfb8d46b90b0408005bff08bfb3b3aad6e458846eec5c23737e28778f94", "DeviceSize": "10737418240" }, "Name": "devicemapper" }, "Mounts": [], "Config": { "Hostname": "29044717920e", "Domainname": "", "User": "", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "Tty": true, "OpenStdin": true, "StdinOnce": false, "Env": [ "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" ], "Cmd": [ "sh" ], "ArgsEscaped": true, "Image": "busybox", "Volumes": null, "WorkingDir": "", "Entrypoint": null, "OnBuild": null, "Labels": {} }, "NetworkSettings": { "Bridge": "", "SandboxID": "ef3cad5d7be81bdd78795d5c9fc257fde14e0a3bbc63f38800b3e5226cabdf1f", "HairpinMode": false, "LinkLocalIPv6Address": "", "LinkLocalIPv6PrefixLen": 0, "Ports": {}, "SandboxKey": "/var/run/docker/netns/ef3cad5d7be8", "SecondaryIPAddresses": null, "SecondaryIPv6Addresses": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "MacAddress": "", "Networks": { "my-overlay": { "IPAMConfig": null, "Links": null, "Aliases": [ "29044717920e" ], "NetworkID": "cc0db79b09b56e2855799debeb08b17d4c570a63f940fa3bea84212695f040e6", "EndpointID": "abdf2d84e506333fab8f6a5b65704b720497aa121965f1a4956e8a852baf18fb", "Gateway": "", "IPAddress": "192.168.3.3", "IPPrefixLen": 24, "IPv6Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "MacAddress": "02:42:c0:a8:03:03", "DriverOpts": null } } } } ] You have new mail in /var/spool/mail/root [root@docker-2 ~]#

root@instance-1:~# docker network inspect my-overlay [ { "Name": "my-overlay", "Id": "cc0db79b09b56e2855799debeb08b17d4c570a63f940fa3bea84212695f040e6", "Created": "2017-12-28T16:59:51.149732661Z", "Scope": "global", "Driver": "overlay", "EnableIPv6": false, "IPAM": { "Driver": "default", "Options": {}, "Config": [ { "Subnet": "192.168.3.0/24" } ] }, "Internal": false, "Attachable": false, "Ingress": false, "ConfigFrom": { "Network": "" }, "ConfigOnly": false, "Containers": { "c34854d53ce7613c3a20f73f4d429b4ec1e3a6a3ba0cc50c257bb471f5184e7e": { "Name": "containerA", "EndpointID": "b178364726482e9444a1dff3acba64d3b241eeab989647610a4f591a4ddbad28", "MacAddress": "02:42:c0:a8:03:02", "IPv4Address": "192.168.3.2/24", "IPv6Address": "" }, "ep-abdf2d84e506333fab8f6a5b65704b720497aa121965f1a4956e8a852baf18fb": { "Name": "containerB", "EndpointID": "abdf2d84e506333fab8f6a5b65704b720497aa121965f1a4956e8a852baf18fb", "MacAddress": "02:42:c0:a8:03:03", "IPv4Address": "192.168.3.3/24", "IPv6Address": "" } }, "Options": {}, "Labels": {} } ]

root@ubuntu-vm-1404:~# docker info Containers: 20 Running: 1 Paused: 0 Stopped: 19 Images: 28 Server Version: 17.06.2-ee-5 Storage Driver: aufs Root Dir: /var/lib/docker/aufs Backing Filesystem: extfs Dirs: 80 Dirperm1 Supported: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: 6e23458c129b551d5c9871e5174f6b1b7f6d1170 runc version: 810190ceaa507aa2727d7ae6f4790c76ec150bd2 init version: 949e6fa Security Options: apparmor Kernel Version: 4.4.0-101-generic Operating System: Ubuntu 14.04.5 LTS OSType: linux Architecture: x86_64 CPUs: 4 Total Memory: 7.796GiB Name: ubuntu-vm-1404 ID: HRHN:J4O2:IBNT:Z65D:HLX6:6DHC:RVJD:BISJ:AHRK:SWAB:WZMN:YUDK Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): false Registry: https://index.docker.io/v1/ Labels: provider=generic Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false

WARNING: No swap limit support root@ubuntu-vm-1404:~#

[Vacant0mens]

Have you checked netstat -ano to see if port 4789/udp (overlay container communication) is open/listening? For me, it seems to be a problem with custom overlay networks not opening that port properly.

ps - adding three ` (back-tick's) before and after your json/terminal output (block code) and one back-tick before and after command-line commands (in-line code) will make them more readable.

[Deequeue]

Have any workarounds / fixes been posted for this issue as yet?

I am also experiencing the same bug in testing, using the stable apt repo for ubuntu.

netstat -ano shows 4789 udp in a "blank" non listening state on both hosts being tested.

[ghost]

@Vacant0mens did you get this fixed?

[Vacant0mens]

I didn't. We stopped using Docker temporarily until some of these network issues get fixed.

I've heard that Windows Server 1709, and 1803 with Docker 18.03 fixes a lot of the networking issues but I haven't yet been able to confirm it myself.

[tomery2000]

Has this problem been solved? I managed to get udp port 4789 open by connecting to my cluster using this command

sudo /usr/bin/dockerd -H tcp://0.0.0.0:4789 -H unix:///var/run/docker.sock --cluster-advertise enp0s3:4789 --cluster-store consul://10.0.2.157:8500

I can see the network as well as add containers to it just like but still cannot ping containers across the network, just like OP.

Might be worth noting, I am behind a corporate proxy. Not sure about anyone else.

[tomery2000]

UPDATE

I managed to solve the issue. Not sure if anyone else has or if we were suffering from different things. I had an error which I had decided to ignore saying something along the lines of "reeeee its got the same name reeeee" turns out this was pretty important and what was breaking it. To solve I ran "nmtui" which gives you a nice blue screen and a menu, I then went to the bottom option and changed the hostname of one of my machines. If machine1 is host and machine 2&3 are the two on the network I changed one of 2 or 3. Anything will do, as long as they are not the same. This solved the issue, if I worded it badly just say and I will try again.

Link to video I used. Only first minute or two are required. https://www.youtube.com/watch?v=i23D7wNCuuc

[ghost]

I got my problems solved by running 18.06.1-ce on both Linux (Ubuntu/Debian) and Windows (server 1803) on physical servers. For Windows, you have to compile 18.06.1-ce yourself.

[ArtoriaRen]

UPDATE

I managed to solve the issue. Not sure if anyone else has or if we were suffering from different things. I had an error which I had decided to ignore saying something along the lines of "reeeee its got the same name reeeee" turns out this was pretty important and what was breaking it. To solve I ran "nmtui" which gives you a nice blue screen and a menu, I then went to the bottom option and changed the hostname of one of my machines. If machine1 is host and machine 2&3 are the two on the network I changed one of 2 or 3. Anything will do, as long as they are not the same. This solved the issue, if I worded it badly just say and I will try again.

Link to video I used. Only first minute or two are required. https://www.youtube.com/watch?v=i23D7wNCuuc

This also solves my problem! Thank you!