hardkernel / linux

Linux kernel source tree
Other
427 stars 408 forks source link

Docker with Traefik running inside swarm manager as a service is not able to redirect requests to services that are in the same overlay network outside from the manager to other service in worker nodes. #301

Closed al-sabr closed 7 years ago

al-sabr commented 7 years ago

My colleagues and I we went in extensive bug tracking and we came up with the following result.

On the Odroid C1 with Docker and the Swarm mode with 2 nodes

  1. Manager master node C1
  2. Worker node C1

the networking communication between the 2 nodes doesn't work.

There is probably a bug in the VXLAN driver or maybe in the kernel. We could not come up with a precise pinpoint where is the problem but one thing certain is that I tried the same settings with 2 Odroid C2 devices and the same setup was working without a glitch.

You can follow the whole bug tracking process in other tickets we created for this specific case:

https://github.com/hypriot/image-builder-odroid-c1/issues/42

https://github.com/moby/moby/issues/33531

al-sabr commented 7 years ago

The main error that was coming from Traefik was the following:

time="2017-06-05T09:18:00Z" level=info msg="Skipping same configuration for provider docker" 
time="2017-06-05T09:18:02Z" level=warning msg="Error forwarding to http://10.0.0.9:80, err: dial tcp 10.0.0.9:80: getsockopt: no route to host" 
time="2017-06-05T09:18:02Z" level=debug msg="Round trip: http://10.0.0.11:9000, code: 200, duration: 821.781684ms"
ardje commented 7 years ago

I have no idea what you are doing.

This might be a problem: If you set up macvlan as low cost bridging between network namespaces, be aware that the master interface can't communicate with any of the slaves. Only slave to slave. So if you have several network namespaces, configure the interface of the host to be a macvlan and do not use the master interface in any way except for creating the macvlans.

This is the only hint I can give, as I have no idea what you are doing, but it sounds like a definite network configuration problem.

al-sabr commented 7 years ago

It's a docker setup with the swarm mode enabled. Docker manages the master and worker nodes automatically.

Docker takes care of the IPTABLES

I just manually created a docker overlay network and overlay networks need the VXLAN kernel option enabled.

The same setup works well with Odroid C2s but not with Odroid C1s

To be able to redirect the traffic the Proxy Traefik is going through docker's own load balancing mechanism. It seems that something is blocking the process in the driver or something like that.

Check this ticket I created for the docker team https://github.com/moby/moby/issues/33531

We think that the bug is inside the driver or the kernel

al-sabr commented 7 years ago

Closed as solution explained in https://github.com/moby/moby/issues/33531