Keepalived Blocks Docker's Overlay Network

[ProjectInitiative]

Setup:

5 Physical nodes

• Node1: 192.168.1.100 - manager
• Node2: 192.168.1.101 - manager
• Node3: 192.168.1.102 - manager
• Node4: 192.168.1.104 - worker
• Node5: 192.168.1.105 - worker

Create Test Overlay Network

docker create network test -d overlay --attachable

Deploy Test Stack

docker stack deploy test-stack --compose-file ./test-stack.yml

Run Tests

First find whichever node ubuntu1 (test server) is running on:

docker stack ps test-stack

Pick any other node and docker exec or ssh into ubuntu2

Running any of the following 5 commands produce a response from the test server:

ping test-stack_ubuntu1

curl test-stack_ubuntu1:8000
curl <ANY-MANAGER-NODE-IP>:8000

telnet test-stack_ubuntu1:8000
telnet <ANY-MANAGER-NODE-IP>:8000

Start osixia/keepalived:2.0.20

Run this on any of the manager nodes:

docker-compose -f ./keepalived.yml up -d

Re-Run Test

Re-running the tests above ping seems to work, but all others fail to connect

keepalived-sample-configs.zip

[ProjectInitiative]

Issue was solved after discovering an IP subnet mask mismatch that propagated throughout the environment. All is working as expected.