Closed BretFisher closed 5 years ago
I am trying out the stack-socat-proxy.ym example and wanted to share what happens when you use the RexRay DigitalOcean driver and deploy mode global. In this example DigitalOcean will create a [stackname]-acme block store drive and the first Traefik container that runs will attached this drive to one of the swarm nodes. Now since we are doing deploy mode global the other Traefik containers will fail to start because they can't share the same drive. The short term solution was to comment out the mode setting and just running one Traefik container. Long term solution is to use a different storage provider that can be shared across all the swarm nodes.
Global mode with a file-based Traefik config is not the correct way to make Traefik highly-available. You'll need a more complex solution with a key/value store, as documented here. There are various issues with the solution of a "shared file endpoint" for Traefik, so a volume won't solve the problem.
Sorry, I copied that socat yaml file in not realizing it had global mode set. As mentioned I hadn't tested it yet :).
Multi-node volumes are not a rexray or docker limitation, but a limit of the storage itself. It's common for most cloud node storage to not support multi-node read+write. Digitalocean Block Storage only supports connecting to one node at a time, as does AWS EBS. If on AWS you could use EFS to share files across multiple nodes with rexray, but this doesn't address Traefik HA like the cluster link above does.
This enhancement GHIssue is about using socat to prevent Traefik running on managers, not multi-node support for Traefik, which I've created as a separate issue here #12
I verified that this setup works:
version: '3.6'
volumes:
acme:
driver: rexray/dobs
driver_opts:
size: 1
networks:
proxy:
external: true
services:
traefik:
image: traefik:1.7-alpine
networks:
- proxy
volumes:
- acme:/etc/traefik/acme
ports:
- 80:80
- 443:443
- 8080:8080 # traefik dashboard
command:
- --Docker
- --Docker.EndPoint=http://dockersocket:2375
- --Docker.SwarmMode
- --Docker.Watch
- --api
# - --defaultentrypoints=http,https
# - --acme
# - --acme.email=[replace_with_email]
# - --acme.httpchallenge
# - --acme.httpchallenge.entrypoint=http
# - --acme.onhostrule=true
# - --acme.entrypoint=https
# - --entryPoints=Name:https Address::443 TLS
# - --entryPoints=Name:http Address::80 Redirect.EntryPoint:https
# - --acme.storage=/etc/traefik/acme/acme.json
# - --acme.acmelogging
# - --acme.caserver=https://acme-v02.api.letsencrypt.org/directory
logging:
options:
max-size: "500k"
deploy:
placement:
constraints:
- node.role == worker
dockersocket:
image: tecnativa/docker-socket-proxy
networks:
- proxy
volumes:
- /var/run/docker.sock:/var/run/docker.sock
environment:
CONTAINERS: 1
NETWORKS: 1
SERVICES: 1
SWARM: 1
TASKS: 1
deploy:
mode: global
placement:
constraints:
- node.role == manager
logging:
options:
max-size: "500k"
I also have the worker nodes behind a DigitalOcean load balancer and if you configure DNS for the load balancer and uncomment the lets encrypt options SSL works too.
Fixed in the big proxy update! 🎉 https://github.com/BretFisher/dogvscat/commit/97a27e1437579c66a228d35c8ae302fdfed9d975
For improved security, we should use socat to offload incoming web traffic hitting the traefik proxies, from being on Swarm managers. Ideally, managers are only "managing" and putting socat in the proxy stack can do that for us.
Mike talks about it in his block post here: https://blog.mikesir87.io/2018/07/letting-traefik-run-on-worker-nodes/
Thanks @pascalandy for the reference: https://github.com/pascalandy/docker-stack-this/blob/master/traefik_stack5/toolproxy.yml
I added a sample stack yaml here but don't think it worked the first time I tried, so needs to have more time testing to see how it should be configured for dogvscat: https://github.com/BretFisher/dogvscat/blob/master/stack-socat-proxy.yml