search

newsnowlabs / docker-ingress-routing-daemon

Docker swarm daemon that modifies ingress mesh routing to expose true client IPs to service containers
MIT License
179 stars 34 forks source link

Questions on running/launching DIRD #8

Closed alexanderadam closed 3 years ago

alexanderadam commented 3 years ago

I'm not quite sure whether this is the right place for this and I hope that it it'll be of no harm if I ask here.

But first of all thank you so much for creating this script!

  1. Will docker-ingress-routing-daemon work if Docker Swarm is using user namespaces?
  2. Migrating on an existing system would mean that the services have to be deleted first (docker service rm …) or scaled down (docker service scale …=0), then run the daemon and then recreate or scale up. Is that correct?
  3. What's the best setup to make it surviving reboots and launch in the correct order?
struanb commented 3 years ago

Hello. Happy to answer your questions as best I can.

  1. I don't know for sure, but don't see why not. I'm not aware that use of user namespaces affect how service networking is implemented.
  2. You are correct.
  3. We use a systemd unit (on systems running systemd) and/or s6 and an s6 service runscript, both to start DIND and to keep it running. Any process supervisor will do, but: on nodes being used as load balancers it is necessary to configure it so that DIND is launched after dockerd; on nodes that are not being used as load balancers that are running service containers, it is only necessary that DIND is launched before dockerd launches service containers (which is a slightly more relaxed criteria).

I hope this helps, good luck with the rest of your DIND setup, and please let me know how it goes.

alexanderadam commented 3 years ago

Wow, thank you for answering so incredibly fast!

Any chance that you could share the systemd service file?

struanb commented 3 years ago

Hi @alexanderadam. Time for a few corrections re my point (3) above:

  1. Having checked our systems, we're using systemd to launch s6, which in turn launches DIND; not DIND directly. Of course systemd could launch DIND directly, at least I see no reason why not. In case it helps, I've copied the systemd unit we're using to launch s6 down below.
  2. On nodes used as load balancers, it is not actually necessary to configure DIND so that it launches after dockerd, as long as the supervisor system you use will keep trying to relaunch DIND if it fails. This work out because DIND itself will fail to start up if it can't talk to dockerd. On our load balancer nodes, the s6 supervisor will repeatedly relaunch DIND (with a delay of 1s) if it fails. This is important not just so that DIND will be launched after dockerd, but also so that if (for any reason) DIND fails later, it will be restarted promptly.
  3. We use the same s6 system to launch DIND on our non-load-balancer nodes too.

Here's the systemd unit we use to launch s6, installed at /etc/systemd/system/s6.service:

[Unit]
Description=S6
After=sysinit.target

[Service]
ExecStart=/usr/bin/s6-svscan /etc/service
Restart=always

[Install]
WantedBy=multi-user.target

Here's our s6 runscript, installed at /etc/service/docker-ingress-routing-daemon/run:

#!/bin/bash

. ./data/ingress-node-gateway-ips.sh

exec /usr/bin/pgrphack ./data/docker-ingress-routing-daemon-v3.1.0 --install >>/var/log/docker-ingress-routing-daemon.log 2>&1

We also have:

I hope this helps.