search

moby / moby

The Moby Project - a collaborative project for the container ecosystem to assemble container-based systems
https://mobyproject.org/
Apache License 2.0
68.53k stars 18.63k forks source link

Windows Swarm containers can't communicate with Swarm node they are running on. #41205

Open mpnewcomb opened 4 years ago

mpnewcomb commented 4 years ago

TL;DR: A Swarm service can't communicate with a process running on the Swarm node the container is running on.

Description: I have a 2 node Swarm: swarm1 and swarm2. I run a simple Golang web server on these nodes. Not as containers, just open up a command prompt and launch the web server. I have a simple Swarm stack with one global service that simply pings itself endlessly. So, each node in the Swarm will have a container running that I can powershell into and issue a Invoke-WebRequest command to try and contact the simple Golang web server from within the container. When I powershell into the container running on swarm1, I can contact the simple Golang web server running on swarm2, but not the one running on swarm1. Likewise, when I powershell into the container running on swarm2, I can contact the simple Golang web server running on swarm1.

hello-server.go:

package main

import (
    "fmt"
    "net/http"
)

func main() {
    http.HandleFunc("/", HelloServer)
    http.ListenAndServe(":8080", nil)
}

func HelloServer(w http.ResponseWriter, r *http.Request) {
    if len(r.URL.Path) > 1 {
        fmt.Fprintf(w, "Hello, %s!", r.URL.Path[1:])
    } else {
        fmt.Fprint(w, "Hello!")
    }
}

swarm_bug.yml:

version: '3.8'

services:
    pinger:
        deploy:
            mode: global
        image: mcr.microsoft.com/windows/servercore:ltsc2019
        command: ["ping", "-t", "localhost"]

Steps to reproduce the issue:

  1. Build the simple Golang web server, copy the exe to each swarm node, start the server on each machine.
  2. Initialize the swarm
  3. Join the other node to the swarm
  4. Deploy the stack: docker stack deploy --compose-file .\swarm_bug.yml swarm_bug
  5. On swarm1: docker ps to get the full name of the container
  6. On swarm1: docker exec -i swarm_bug_pinger.<random_id> powershell
  7. On swarm1, inside container: Invoke-WebRequest -URI "http://swarm1:8080" -UseBasicParsing (should fail)
  8. On swarm1, inside container: Invoke-WebRequest -URI "http://swarm2:8080" -UseBasicParsing (should pass)
  9. On swarm2: docker ps to get the full name of the container
  10. On swarm2: docker exec -i swarm_bug_pinger.<random_id> powershell
  11. On swarm2, inside container: Invoke-WebRequest -URI "http://swarm1:8080" -UseBasicParsing (should pass)
  12. On swarm2, inside container: Invoke-WebRequest -URI "http://swarm2:8080" -UseBasicParsing (should fail)

Describe the results you received: Results from swarm1:

PS C:\temp\swarm_bug> docker node ls
ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
yekfp7jy5y82ks65sgmpxcdgb *   swarm1              Ready               Active              Reachable           19.03.8
plaurejtbf5kec2jlr365tev8     swarm2              Ready               Active              Leader              19.03.8

PS C:\temp\swarm_bug> docker stack deploy --compose-file .\swarm_bug.yml swarm_bug
Creating network swarm_bug_default
Creating service swarm_bug_pinger

PS C:\temp\swarm_bug> docker ps
CONTAINER ID        IMAGE                                           COMMAND                  CREATED             STATUS              PORTS               NAMES
84b44fcc4202        mcr.microsoft.com/windows/servercore:ltsc2019   "ping -t localhost"      16 seconds ago      Up 14 seconds                           swarm_bug_pinger.yekfp7jy5y82ks65sgmpxcdgb.osjh4h5zsqxg9pjb2vdd1j63h

PS C:\temp\swarm_bug> docker exec -i swarm_bug_pinger.yekfp7jy5y82ks65sgmpxcdgb.osjh4h5zsqxg9pjb2vdd1j63h powershell
Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.

PS C:\> Invoke-WebRequest -URI "http://swarm2:8080" -UseBasicParsing
Invoke-WebRequest -URI "http://swarm2:8080" -UseBasicParsing

StatusCode        : 200
StatusDescription : OK
Content           : Hello!
RawContent        : HTTP/1.1 200 OK
                    Content-Length: 6
                    Content-Type: text/plain; charset=utf-8
                    Date: Mon, 13 Jul 2020 18:48:10 GMT

                    Hello!
Forms             :
Headers           : {[Content-Length, 6], [Content-Type, text/plain;
                    charset=utf-8], [Date, Mon, 13 Jul 2020 18:48:10 GMT]}
Images            : {}
InputFields       : {}
Links             : {}
ParsedHtml        :
RawContentLength  : 6

PS C:\> Invoke-WebRequest -URI "http://swarm1:8080" -UseBasicParsing
Invoke-WebRequest -URI "http://swarm1:8080" -UseBasicParsing
Invoke-WebRequest : Unable to connect to the remote server
At line:1 char:1
+ Invoke-WebRequest -URI "http://swarm1:8080" -UseBasicParsing
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:Htt
   pWebRequest) [Invoke-WebRequest], WebException
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShe
   ll.Commands.InvokeWebRequestCommand

Results from swarm2:

PS C:\temp> docker node ls
ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
yekfp7jy5y82ks65sgmpxcdgb     swarm1              Ready               Active              Reachable           19.03.8
plaurejtbf5kec2jlr365tev8 *   swarm2              Ready               Active              Leader              19.03.8

PS C:\temp> docker ps
CONTAINER ID        IMAGE                                           COMMAND                  CREATED             STATUS              PORTS               NAMES
856df9e3258b        mcr.microsoft.com/windows/servercore:ltsc2019   "ping -t localhost"      3 minutes ago       Up 3 minutes                            swarm_bug_pinger.plaurejtbf5kec2jlr365tev8.m7ewy9cw6mjbf1k53jk0xn08b

PS C:\temp> docker exec -i swarm_bug_pinger.plaurejtbf5kec2jlr365tev8.m7ewy9cw6mjbf1k53jk0xn08b powershell
Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.

PS C:\> Invoke-WebRequest -URI "http://swarm1:8080" -UseBasicParsing
Invoke-WebRequest -URI "http://swarm1:8080" -UseBasicParsing

StatusCode        : 200
StatusDescription : OK
Content           : Hello!
RawContent        : HTTP/1.1 200 OK
                    Content-Length: 6
                    Content-Type: text/plain; charset=utf-8
                    Date: Mon, 13 Jul 2020 18:51:08 GMT

                    Hello!
Forms             :
Headers           : {[Content-Length, 6], [Content-Type, text/plain;
                    charset=utf-8], [Date, Mon, 13 Jul 2020 18:51:08 GMT]}
Images            : {}
InputFields       : {}
Links             : {}
ParsedHtml        :
RawContentLength  : 6

PS C:\> Invoke-WebRequest -URI "http://swarm2:8080" -UseBasicParsing
Invoke-WebRequest -URI "http://swarm2:8080" -UseBasicParsing
Invoke-WebRequest : Unable to connect to the remote server
At line:1 char:1
+ Invoke-WebRequest -URI "http://swarm2:8080" -UseBasicParsing
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:Htt
   pWebRequest) [Invoke-WebRequest], WebException
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShe
   ll.Commands.InvokeWebRequestCommand

Describe the results you expected: I don't see any valid reason that a container couldn't communicate with the Swarm node it is running on. It should be just another external connection.

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Client: Docker Engine - Enterprise
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        5ecd622f3f
 Built:             05/27/2020 20:32:22
 OS/Arch:           windows/amd64
 Experimental:      false

Server: Docker Engine - Enterprise
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.24)
  Go version:       go1.12.17
  Git commit:       5ecd622f3f
  Built:            05/27/2020 20:31:03
  OS/Arch:          windows/amd64
  Experimental:     false

Output of docker info:

Client:
 Debug Mode: false

Server:
 Containers: 22
  Running: 7
  Paused: 0
  Stopped: 15
 Images: 306
 Server Version: 19.03.8
 Storage Driver: windowsfilter
  Windows:
 Logging Driver: json-file
 Plugins:
  Volume: local
  Network: ics internal l2bridge l2tunnel nat null overlay private transparent
  Log: awslogs etwlogs fluentd gcplogs gelf json-file local logentries splunk syslog
 Swarm: active
  NodeID: yekfp7jy5y82ks65sgmpxcdgb
  Is Manager: true
  ClusterID: cjbn9h403asg60dqrxz3nlnn9
  Managers: 2
  Nodes: 2
  Default Address Pool: 10.0.0.0/8
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 192.168.120.194
  Manager Addresses:
   192.168.120.194:2377
   192.168.120.196:2377
 Default Isolation: process
 Kernel Version: 10.0 17763 (17763.1.amd64fre.rs5_release.180914-1434)
 Operating System: Windows Server 2019 Standard Version 1809 (OS Build 17763.1158)
 OSType: windows
 Architecture: x86_64
 CPUs: 8
 Total Memory: 24GiB
 Name: swarm1
 ID: J6DI:KDDF:5DGJ:3I66:7PBN:GFYW:CYGD:AH56:RS27:MEDD:PZ7M:PEWF
 Docker Root Dir: C:\ProgramData\docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
  com.docker.security.fips=enabled
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: API is accessible on http://localhost:2375 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/engine/security/security/#docker-daemon-attack-surface
WARNING: Running Swarm in a two-manager configuration. This configuration provides
         no fault tolerance, and poses a high risk to lose control over the cluster.
         Refer to https://docs.docker.com/engine/swarm/admin_guide/ to configure the
         Swarm for fault-tolerance.

Additional environment details (AWS, VirtualBox, physical, etc.):

thaJeztah commented 4 years ago

/cc @msabansal