Non-deterministic service discovery when stacks join a shared network

[agunnarsson]

Description

We have a setup with multiple application stacks and one stack of shared services. The application stacks have similar service setup, so service names are the same in each of them. Each application stack have their own overlay network, and some services also join an overlay network from the shared stack. The problem we are facing is that services that join the shared network might connect to a service from another application stack when it is supposed to call a service in the local stack.

Steps to reproduce the issue:

1. Define stacks

shared.yml

version: '3.7'
services:
  sharedservice:
    image: alpine:3.8
    networks:
      - sharednet
    command: ["tail", "-f", "/etc/alpine-release"]

networks:
  sharednet:
    driver: overlay

app.yml

version: '3.7'
services:
  app:
    image: alpine:3.8
    networks:
      - appnet
      - shared_sharednet
    command: ["tail", "-f", "/etc/alpine-release"]

networks:
  appnet:
    driver: overlay
  shared_sharednet:
    external: true

2. Deploy stacks

   
   $ docker stack deploy -c shared.yml shared
   Creating network shared_sharednet
   Creating service shared_sharedservice

$ docker stack deploy -c app.yml app Creating network app_appnet Creating service app_app

$ docker stack deploy -c app.yml xapp Creating network xapp_appnet Creating service xapp_app

3. List containers

$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 9a4f318942db alpine:3.8 "tail -f /etc/alpine…" About a minute ago Up About a minute xapp_app.1.vxe7325a9t7oqzeftyrspd2dw 25bfe424ba8a alpine:3.8 "tail -f /etc/alpine…" About a minute ago Up About a minute app_app.1.qbistswbr7tb3ba1b2z0637e8 508aa06c4239 alpine:3.8 "tail -f /etc/alpine…" About a minute ago Up About a minute shared_sharedservice.1.sux3binf4y7gc1arsz5um2kma

4. Lookup the service app from each container

$ docker exec -it 9a4f318942db nslookup app nslookup: can't resolve '(null)': Name does not resolve

Name: app Address 1: 172.20.158.7 Address 2: 172.20.158.5

$ docker exec -it 25bfe424ba8a nslookup app nslookup: can't resolve '(null)': Name does not resolve

Name: app Address 1: 172.20.160.2

$ docker exec -it 508aa06c4239 nslookup app nslookup: can't resolve '(null)': Name does not resolve

Name: app Address 1: 172.20.158.5 Address 2: 172.20.158.7

**Describe the results you received:**
Service app_app only find the local service app, while services xapp_app and shared_sharedservice find both services named app with ips from shared_sharednet.

**Describe the results you expected:**
I expected app_app and xapp_app to only find the service app from the local stack namespace. I also expected that I would need to add the stack prefix (app_ or xapp_) to find a service on another stack.

**Additional information you deem important (e.g. issue happens only occasionally):**
When inspecting the app tasks I noticed that networks were sorted in different order. Maybe that is the reason why results between app and xapp differ.
Another consequence of this issue is that if a service goes down in the app stack which seems to be working properly, clients in the same stack will start using services with the same name in other stacks connected to the shared network.
Issue is seen on both Ubuntu 18.04 and CentOS 7 hosts with docker versions 18.09.1 to 19.03.4
**Output of `docker version`:**

Client: Docker Engine - Community Version: 19.03.4 API version: 1.40 Go version: go1.12.10 Git commit: 9013bf583a Built: Fri Oct 18 15:54:09 2019 OS/Arch: linux/amd64 Experimental: false

Server: Docker Engine - Community Engine: Version: 19.03.4 API version: 1.40 (minimum version 1.12) Go version: go1.12.10 Git commit: 9013bf583a Built: Fri Oct 18 15:52:40 2019 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.2.10 GitCommit: b34a5c8af56e510852c35414db4c1f4fa6172339 runc: Version: 1.0.0-rc8+dev GitCommit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657 docker-init: Version: 0.18.0 GitCommit: fec3683

**Output of `docker info`:**

Client: Debug Mode: false

Server: Containers: 17 Running: 3 Paused: 0 Stopped: 14 Images: 811 Server Version: 19.03.4 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: active NodeID: 8krhccmxqddhjht49c6wxd612 Is Manager: true ClusterID: b3h50n39yhif63i3ax3w9v53p Managers: 1 Nodes: 1 Default Address Pool: 172.20.128.0/17
SubnetSize: 23 Data Path Port: 4789 Orchestration: Task History Retention Limit: 5 Raft: Snapshot Interval: 10000 Number of Old Snapshots to Retain: 0 Heartbeat Tick: 1 Election Tick: 10 Dispatcher: Heartbeat Period: 5 seconds CA Configuration: Expiry Duration: 3 months Force Rotate: 0 Autolock Managers: false Root Rotation In Progress: false Node Address: 127.0.0.1 Manager Addresses: 127.0.0.1:2377 Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657 init version: fec3683 Security Options: apparmor seccomp Profile: default Kernel Version: 4.15.0-66-generic Operating System: Ubuntu 18.04.3 LTS OSType: linux Architecture: x86_64 CPUs: 8 Total Memory: 30.85GiB Name: narra ID: ZMP5:HH5J:XBFJ:QTCD:BPTH:RKOH:AOBF:4SSF:VDLB:E4KW:72ZO:OKSV Docker Root Dir: /var/lib/docker Debug Mode: false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false

WARNING: No swap limit support

**Additional environment details (AWS, VirtualBox, physical, etc.):**
Bare metal

[agunnarsson]

Maybe it is an odd use case to have a stack of shared resources? At the same time it feels unnecessary to set up external service discovery when swarm comes so close. If resolving services in other stacks without stack prefix is a feature, maybe it should be configurable to turn it off when joining networks?

[cpuguy83]

Wouldn't the services names be namespaced by the app name?

[agunnarsson]

Are you referring to the stack name "app", and that it can be used as prefix for the service name like "stack_service"? That works perfectly, but we have a setup where multiple similar stacks are setup from a common .yml

version: '3.7'
services:
  service1:
...
  client1:
    environment:
      - SERVICE_URL=http://service1

To provide service names including prefix to a client within the stack we would need to create overrides for each deployment to connect services to each other.

version: '3.7'
services:
  client1:
    environment:
      - SERVICE_URL=http://stack1_service1

It would be more convenient to keep the connections in the common .yml since connections are the same in every stack. If there is a way to use the stack name as a variable within .yml files it could be an option, but in our case it would be better to rely on that services in other stacks only resolved when prefix is used.

Finally the confusing part of this issue is that discovery is depending on stack names.

I realize the name "app" cause a lot of confusion here, should have chosen unique names. :-)