Swarm Mode + expose TCP port (ex: ssh) -- crashes haproxy

[ventz]

It looks like currently haproxy crashes if you combine swarm mode + TCP port

Using something like gogs for the example since it provides both the web service + ssh service --it exposes tcp/3000 as "web" and let's say we configure tcp/2222 as "ssh", and the setup being:

# NETWORK:
docker network create --driver overlay proxy

EDIT: Ports need to be flipped to:
-p 3000:80
-p 8022:2222

# HAProxy:
docker service create --name=haproxy \
--network proxy \
-p 3000:80\
-p 8022:2222 \
-e STATS_AUTH="auth:auth" \
-e STATS_PORT=1936 \
-p 1936:1936 \
--constraint "node.role == manager" \
--mount type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock \
dockercloud/haproxy

# GOGS (or other Web+SSH service)
docker service create --name=gogs \
--network proxy \
-e SERVICE_PORTS="3000" \
-e TCP_PORTS="8022" \
--constraint "node.role != manager" \
--mount type=bind,src=/docker/SHARED-DATA/gogs,dst=/data \
gogs/gogs

Any recomendations/suggestions for ways around this?

I've even tried -ADDITIONAL_BACKENDS, which doesn't seem to get picked up (any working examples on -ADITIONAL_BACKENDS would be nice by the way, the manual provides the list format, but not an example)

Thanks.

EDIT: Updated 2222 to 8022 for the internal side, just to differentiate the two in the example. EDIT: Flipped ports, and marked that they need to be flipped.

[tifayuki]

On my own test(dockercloud/haproxy:1.6.2) using tutum/wordpress, I didn't see any crashes.

The log shows as below, which is as expected:

listen port_3306
  bind :3306
  mode tcp
  server wp.1.6880g7viebr54lnat6rfklr9x 10.0.0.3:3306 check inter 2000 rise 2 fall 3
frontend default_port_80
  bind :80
  reqadd X-Forwarded-Proto:\ http
  maxconn 4096
  default_backend default_service
backend default_service
  server wp.1.6880g7viebr54lnat6rfklr9x 10.0.0.3:80 check inter 2000 rise 2 fall 3

What is the error message did you receive?

BTW, to make it work ,you have to specify SERVICE_PORTS="3000, 2222"

[ventz]

@tifayuki

I found one error on my side -> updated (and placed an EDIT on the original) - ports need to be flipped, the service port is the 2nd item.

I have tried 2 config options, and each produces it's own error/problem: (note: I am changing the "internal" tcp/2222 to tcp/8022, just to differentiate that vs the published port)

1.) The original post:

SERVICE_PORTS="3000"
TCP_PORTS="8022"

That produces:

listen port_8022
  bind :8022
  mode tcp
frontend default_port_80
  bind :80
  reqadd X-Forwarded-Proto:\ http
  maxconn 4096
  default_backend default_service
backend default_service
  server gogs.1.cxqrjldsfrow5vbo5zwjud1uu 10.0.0.5:3000 check inter 2000 rise 2 fall 3

Issue with that: no server for listen port_8022

2.) Also tried later:

SERVICE_PORTS="3000, 8022"
# REMOVED by comment out #: TCP_PORTS="8022"

That produces:

frontend default_port_80
  bind :80
  reqadd X-Forwarded-Proto:\ http
  maxconn 4096
  default_backend default_service
backend default_service
  server gogs.1.685k0e9hqtgmvtzc70iv712p7 10.0.0.5:3000 check inter 2000 rise 2 fall 3
  server gogs.1.685k0e9hqtgmvtzc70iv712p7 10.0.0.5:8022 check inter 2000 rise 2 fall 3

Issue with that: both of the backends are treated like http, and no listen for the ssh port

From your last line, I am assuming you mean the tcp port needs to be in SERVICE_PORTS and TCP_PORTS?

Example:

SERVICE_PORTS="3000, 8022"
TCP_PORTS="8022"

Just got back and tried using both -- that produces the correctly generated config:

listen port_8022
  bind :8022
  mode tcp
  server gogs.1.2ddc13md73j11uioxdcduknna 10.0.0.5:8022 check inter 2000 rise 2 fall 3
frontend default_port_80
  bind :80
  reqadd X-Forwarded-Proto:\ http
  maxconn 4096
  default_backend default_service
backend default_service
  server gogs.1.2ddc13md73j11uioxdcduknna 10.0.0.5:3000 check inter 2000 rise 2 fall 3

I think that's the key part -- that "SERVICE_PORTS" needs to contain ALL ports, even though the tcp ports are not "http" ports, and they are in the TCP_PORTS config.

Thanks for that last tid bit! (If you can - would you mind updating the README with an example -- for either mysql or ssh, etc, just so that it's clear to others)

[tifayuki]

In the current implementation of swarmkit, there is no way to read a container/service' exposed ports through swarm api. Therefore haproxy script doesn't know which port to use for the load balancing. And this is why we introduce the environment variable SERVICE_PORTS as a workaround. You can simply put all the exposed ports of you application container as the value to make it work.

I will try to update with an example to clarify how to use it.