Docker swarm container cannot connect service port in other container

Description

I'm running 3 service in 2 nodes in a docker swarm using docker stack deploy a yaml file. Here' my docker version:

docker version
Client:
 Version:      1.13.1
 API version:  1.26
 Go version:   go1.7.5
 Git commit:   092cba3
 Built:        Wed Feb  8 06:38:28 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.13.1
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   092cba3
 Built:        Wed Feb  8 06:38:28 2017
 OS/Arch:      linux/amd64
 Experimental: false

I'm using the default overlay network for swarm.

docker network inspect vikro_keke_default
[
    {
        "Name": "vikro_keke_default",
        "Id": "i5dub8ikcz750trpce0wdbu9g",
        "Created": "2017-03-06T16:28:53.258523288+08:00",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.0.0/24",
                    "Gateway": "10.0.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Containers": {
            "51f9bebad132592a83d52490ab41fb7f7c51e560aaea35470bdba4077e884f54": {
                "Name": "vikro_keke_rabbitmq_keke.1.tnvgpz75mmkrzzc6zv2kzj4l0",
                "EndpointID": "147e8682aae09df2a250afb988bffa2b42dc1293fd5c879eeae1e32dbc89e7fc",
                "MacAddress": "02:42:0a:00:00:03",
                "IPv4Address": "10.0.0.3/24",
                "IPv6Address": ""
            },
            "bc880c77a8e5586b2a5fdffb4d76ee47aa57e226b854961e96014cf386f1431d": {
                "Name": "vikro_keke_math.1.a26tv4f60e2v6vbn5l58tqms9",
                "EndpointID": "fd8dfb96314eb58205a68918e5473a86aa13939c3664bae79db5539689996cd8",
                "MacAddress": "02:42:0a:00:00:07",
                "IPv4Address": "10.0.0.7/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4097"
        },
        "Labels": {
            "com.docker.stack.namespace": "vikro_keke"
        },
        "Peers": [
            {
                "Name": "LocalVM-b21c412a5aef",
                "IP": "10.21.100.145"
            },
            {
                "Name": "Keke-Ubuntu-0d7dc23057aa",
                "IP": "10.21.22.165"
            }
        ]
    }
]

I can ping services using name but I failed to connect to the service port inside container. For example: When I ping rabbitmq_keke, I got 10.0.0.2 while in inspect it says 10.0.0.3, but I cannot telnet the 5672 and 15672 port from other container, while these ports can be accessed outside container.

Here's the yaml file I use to start stack:

version: "3"

services:

  rabbitmq_keke:
    image: 10.21.22.165:5000/rabbitmq_keke:0.2
    ports:
      - 5672:5672
      - 15672:15672
    deploy:
      placement:
        constraints: [node.role == manager]

  hello:
    image: 10.21.22.165:5000/vikro:0.2
    links:
      - rabbitmq_keke
    ports:
      - 8088:8088
    command: hello.HelloService
    deploy:
      replicas: 1

Steps to reproduce the issue:

Start swarm service using yaml file
Attach to container and telnet the service port
Failed to access the port

Describe the results you received: I can ping the name but cannot telnet the service port

Describe the results you expected: Container can access service port

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Client:
 Version:      1.13.1
 API version:  1.26
 Go version:   go1.7.5
 Git commit:   092cba3
 Built:        Wed Feb  8 06:38:28 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.13.1
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   092cba3
 Built:        Wed Feb  8 06:38:28 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 48
 Running: 2
 Paused: 0
 Stopped: 46
Images: 29
Server Version: 1.13.1
Storage Driver: devicemapper
 Pool Name: docker-253:0-1154740-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 2.354 GB
 Data Space Total: 107.4 GB
 Data Space Available: 46.01 GB
 Metadata Space Used: 6.459 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.141 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.135-RHEL7 (2016-11-16)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: active
 NodeID: 5qfgo2rcl5grn4xi5uj9c0vew
 Is Manager: true
 ClusterID: xrxeojzf5xb6a6mm3u9nyz4zo
 Managers: 1
 Nodes: 2
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: 10.21.100.145
 Manager Addresses:
  10.21.100.145:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1
runc version: 9df8b306d01f59d3a8029be411de015b7304dd8f
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-514.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.797 GiB
Name: LocalVM
ID: APMI:MS2X:MYGR:WZX2:MXHW:X2NR:IYDD:X4GE:6F57:LW64:RSS6:JSFX
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-ip6tables is disabled
Labels:
 provider=generic
Experimental: false
Insecure Registries:
 10.21.22.165:5000
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.): One of the swarm node (Ubuntu) was running in a VirtualBox while the other (Centos) is running in a physical machine.

moby / moby

Docker swarm container cannot connect service port in other container #31565