Cannot be found error When NAT'ing to services on SWARM event stream

[BrianAdams]

Accessing the service on the load balanced port fails. Using GoRB+GoRB Docker link on a SWARM cluster named "swarm-master". For this example, all of the services are running on the same node swarm-node-1. Trying to run a simple service on port 8000 but it fails. Using the docker images, kobolog/gorb in docker hub as of today. The only thing different from a standard deployment is that we are listening to the swarm event stream instead of the local HOST socket. If I use the HOST socket for the docker-link then it works correctly.

GoRB service

docker $(docker-machine config swarm-node-1) run --net=host --rm --privileged kobolog/gorb -f -i eth0

GoRB Docker Link

docker $(docker-machine config swarm-node-1) run --net=host --rm -e DOCKER_HOST=$(docker-machine env --swarm swarm-master | grep DOCKER_HOST | awk 'BEGIN { FS = "=" } ; { print $2}' | tr -d '"') -e DOCKER_TLS_VERIFY="1" -e DOCKER_CERT_PATH="/etc/docker/" -v /etc/docker/server-key.pem:/etc/docker/key.pem:ro -v /etc/docker/server.pem:/etc/docker/cert.pem:ro -v /etc/docker/ca.pem:/etc/docker/ca.pem:ro  kobolog/gorb-docker-link -r 172.17.0.1:4672 -i eth0

And running a container

docker $(docker-machine config swarm-node-1) run -d --name motd -p 8000 dockhero/motd-http

Executing:

curl -sS $(docker-machine ip swarm-node-1):8000

Expect:

From listening comes wisdom and from speaking repentance.

But Got:

curl: (7) Failed to connect to x.x.x.x port 8000: Connection refused

Logs from GoRB Server

time="2015-12-09T20:33:35Z" level=info msg="starting GORB Daemon v0.2" time="2015-12-09T20:33:35Z" level=info msg="initializing IPVS context" time="2015-12-09T20:33:35Z" level=info msg="setting up HTTP server on :4672" time="2015-12-09T21:46:47Z" level=info msg="creating virtual service [dockhero-motd-http_8000_tcp] on 104.236.147.174:8000"

Logs from GoRB Linker

time="2015-12-09T21:46:47Z" level=info msg="creating [dockhero-motd-http_8000_tcp/swarm-node-1/motd_8000_tcp] with 104.236.147.174:32768 -> 8000" time="2015-12-09T21:46:47Z" level=info msg="creating service [dockhero-motd-http_8000_tcp] on port 8000/tcp" time="2015-12-09T21:46:47Z" level=warning msg="no public ports were processed for [dockhero/motd-http/swarm-node-1/motd]"

...

Time="2015-12-09T21:09:56Z" level=warning msg="errors while exposing existing containers: [service parent [dockhero-motd-http_8000_tcp] cannot be found

and after removing the motd container...

time="2015-12-09T21:56:27Z" level=info msg="removing [dockhero-motd-http_8000_tcp/motd_8000_tcp] with 104.236.147.174:32768 -> 8000" time="2015-12-09T21:56:27Z" level=warning msg="no public ports were processed for [dockhero/motd-http/motd]" time="2015-12-09T21:56:27Z" level=error msg="error(s) while processing container 42d598b98522294c9060dcfdced9b737320bc2650649ce1ee89803dfcb9aa921: [backend [dockhero-motd-http_8000_tcp/motd_8000_tcp] cannot be found]"

Query of the service from GoRB:

curl -sS 127.0.0.1:4672/service/dockhero-motd-http_8000_tcp

{ "options": { "host": "", "port": 8000, "protocol": "tcp", "method": "wrr", "persistent": false }, "health": 1, "backends": null }

The Docker Hosts network config:

root@swarm-node-1:~# ifconfig docker0 Link encap:Ethernet HWaddr 02:42:5a:cd:6d:f2 inet addr:172.17.0.1 Bcast:0.0.0.0 Mask:255.255.0.0 inet6 addr: fe80::42:5aff:fecd:6df2/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:479175 errors:0 dropped:0 overruns:0 frame:0 TX packets:801946 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:61015167 (58.1 MiB) TX bytes:110121273 (105.0 MiB)

docker_gwbridge Link encap:Ethernet HWaddr 02:42:34:80:f9:5e inet addr:172.18.0.1 Bcast:0.0.0.0 Mask:255.255.0.0 inet6 addr: fe80::42:34ff:fe80:f95e/64 Scope:Link UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:13 errors:0 dropped:0 overruns:0 frame:0 TX packets:15 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:956 (956.0 B) TX bytes:1182 (1.1 KiB)

eth0 Link encap:Ethernet HWaddr 04:01:8d:fc:d1:01 inet addr:104.236.147.174 Bcast:104.236.191.255 Mask:255.255.192.0 inet6 addr: fe80::601:8dff:fefc:d101/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:906074 errors:0 dropped:0 overruns:0 frame:0 TX packets:524341 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:662057540 (631.3 MiB) TX bytes:74092490 (70.6 MiB)

[BrianAdams]

Hopefully obvious, but the Gorp service when queried on a normally working instance is:

{ "options": { "host": "", "port": 8000, "protocol": "tcp", "method": "wrr", "persistent": false }, "health": 1, "backends": [ "motd_8000_tcp" ] }

Where the variance is in the backends

[BrianAdams]

Looks like this may be similar to the last naming issue. When the services are listed from the swarm event stream they have an additional data element which is the host they are running on.

Swarm Stream: dockhero-motd-http_8000_tcp/swarm-node-1/motd_8000_tcp] Host Stream: dockhero-motd-http_8000_tcp/motd_8000_tcp