Let me start by saying that I am not convinced that this is a docker swarm issue, although all investigation so far points in this direction.

I have a couple of services running in a docker swarm on a single docker host. All services run in the same overlay network. These services all expose a different port on which a web server is available. The docker-host runs CoreOS (1520.0.0 Alpha channel).

Sometimes I end up in a situation in which requests made on http://docker-host.local: timeout. When I login on the docker-host and make a request to localhost: it also times out. However from a shell in a different container a request to the service does succeed without issues.

docker service ls shows the correct port mappings.

The service that is not reachable, is seemingly random. Sometimes all are functioning correctly, sometimes one is not reachable, sometimes it resolves after some time.

I have inspected the docker networks, they do not conflict the with the host network.

I can reproduce this by creating a stack of nginx services, hosting the default webpage. file: docker-compose-test.yml

version: '3.1'
services:
  nginx1:
    image: nginx:1.11.8-alpine
    networks:
      - test
    ports:
      - "10081:80"
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure

  nginx2:
    image: nginx:1.11.8-alpine
    networks:
      - test
    ports:
      - "10082:80"
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure

  nginx3:
    image: nginx:1.11.8-alpine
    networks:
      - test
    ports:
      - "10083:80"
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure

  nginx4:
    image: nginx:1.11.8-alpine
    networks:
      - test
    ports:
      - "10084:80"
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure

  nginx5:
    image: nginx:1.11.8-alpine
    networks:
      - test
    ports:
      - "10085:80"
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure

  nginx6:
    image: nginx:1.11.8-alpine
    networks:
      - test
    ports:
      - "10086:80"
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure

  nginx7:
    image: nginx:1.11.8-alpine
    networks:
      - test
    ports:
      - "10087:80"
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure

  nginx8:
    image: nginx:1.11.8-alpine
    networks:
      - test
    ports:
      - "10088:80"
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure

  nginx9:
    image: nginx:1.11.8-alpine
    networks:
      - test
    ports:
      - "10089:80"
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
networks:
  test:

This script will deploy the stack, test availability and take down the stack until the error situation is reached. file: test-docker-swarm.sh

#!/bin/bash

DOCKER_HOST=$1
fail=0

while [[ ${fail} -eq 0 ]] ; do
  docker -H ${DOCKER_HOST} stack deploy -c docker-compose-test.yml test
  sleep 15

  for i in $(seq 1 9) ; do
    request="http://${DOCKER_HOST}:1008${i}"
    echo "making request: ${request}"
    curl -s -o /dev/null --max-time 2 ${request}
    if [[ $? -ne 0 ]] ; then
        echo request failed: ${request}
        fail=1
    fi
  done

  if [[ ${fail} -eq 0 ]] ; then
      docker -H ${DOCKER_HOST} stack down test

    while [[ $(docker -H ${DOCKER_HOST} network ls --filter 'name=^test_' | wc -l) -ne 1 ]]; do
      echo "waiting for stack to go down"
      sleep 2
    done
  fi
done

execute running: `./test-docker-swarm.sh

I have no clue what steps I can take to debug, and resolve this. Any pointers are appreciated.

docker version

Client:
 Version:      17.06.1-ce
 API version:  1.30
 Go version:   go1.8.2
 Git commit:   874a737
 Built:        Tue Aug 29 23:50:27 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.06.1-ce
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.2
 Git commit:   874a737
 Built:        Tue Aug 29 23:50:09 2017
 OS/Arch:      linux/amd64
 Experimental: false
`
#docker info
`
Containers: 9
 Running: 9
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 17.06.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: x06mlhlwqyo3dg4lmigy18z1q
 Is Manager: true
 ClusterID: qy022nd3bjn1157sxcc6qzr9n
 Managers: 1
 Nodes: 1
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Root Rotation In Progress: false
 Node Address: 10.255.11.40
 Manager Addresses:
  10.255.11.40:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 6e23458c129b551d5c9871e5174f6b1b7f6d1170
runc version: 810190ceaa507aa2727d7ae6f4790c76ec150bd2
init version: v0.13.2 (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 seccomp
  Profile: default
 selinux
Kernel Version: 4.13.0-rc7-coreos
Operating System: Container Linux by CoreOS 1520.0.0 (Ladybug)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 5.776GiB
Name: fqfs-development
ID: RCNI:3ZUR:LTDA:ABIB:EYEW:HCIY:H2RC:XDNT:LC77:BMQH:FKXI:T6YZ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

docker-archive / classicswarm

Communication to containers not always possible #2794

docker version