Requests fail in Windows Server 2016 swarm overlay network when using service virtual IP or hostname

StetsonG commented 6 years ago

I think this is the correct repository for this issue (Docker Swarm on Windows Server 2016) but please let me know if there is a better place for it.

Expected behavior

An HTTP GET request succeeds against a swarm service's virtual IP or hostname from another container on the same overlay network.

Actual behavior

An HTTP GET request succeeds against the real container IP address, but not against the service's virtual IP or hostname.

Information

Running in Windows Server 2016 version 1607

PS C:\Windows\system32> docker version
Client:
 Version:      17.06.1-ee-2
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   8e43158
 Built:        Wed Aug 23 21:16:53 2017
 OS/Arch:      windows/amd64

Server:
 Version:      17.06.1-ee-2
 API version:  1.30 (minimum version 1.24)
 Go version:   go1.8.3
 Git commit:   8e43158
 Built:        Wed Aug 23 21:25:53 2017
 OS/Arch:      windows/amd64
 Experimental: false

The swarm consists of three Windows Server 2016 hosts, but I can reproduce this issue when constraining all services/containers to a single node.

Note: I originally experienced this issue in a deployment with multiple asp net core apps in a stack deployment, but have reproduced this problem with a single manually deployed service and powershell container on the same overlay network.

Note: Calling services by their hostnames does work when using docker-compose up instead of docker service create. In this case there is no virtual IP so the hostname resolves directly to the container IP and the request succeeds.

Steps to reproduce the behavior

Download and build the aspnetcore sample docker container. This is just the simplest testable web app I could find that runs on Windows Containers. My actual applications are also asp.net core 2.0 applications.

https://github.com/dotnet/dotnet-docker-samples/tree/master/aspnetapp

Create an overlay network: > docker network create --driver overlay --attachable testoverlay
1. Create a service with the aspnetcore sample container, publish port 80: > docker service create --constraint 'node.hostname == ac-devcontnr-01' --network testoverlay --publish mode=host,target=80,published=80 --name aspnetapp aspnetapp

I can then go to http://[dockerhost]:80 and I can see the asp net core sample app.

Create a powershell container on the same overlay network to test service requests: > docker run -it --network testoverlay microsoft/powershell:nanoserver
Test web request to the aspnetapp container ip: Succeeds

PS C:\> Invoke-Webrequest 10.0.0.3

StatusCode        : 200
StatusDescription : OK
Content           : <!DOCTYPE html>
                    <html>
                    <head>
                        <meta charset="utf-8" />
                        <meta name="viewport" content="width=device-width, initial-scale=1.0" />
                        <title>Home Page - aspnetapp</title>

                       ...
RawContent        : HTTP/1.1 200 OK
                    Date: Tue, 17 Oct 2017 15:39:09 GMT
                    Transfer-Encoding: chunked
                    Server: Kestrel
                    Content-Type: text/html; charset=utf-8

                    <!DOCTYPE html>
                    <html>
                    <head>
                        <meta charset="utf-8" ...
Forms             :
Headers           : {[Date, System.String[]], [Transfer-Encoding, System.String[]], [Server, System.String[]], [Content-Type, System.String[]]}
Images            : {@{outerHTML=<img src="/images/banner1.svg" alt="ASP.NET" class="img-responsive" />; tagName=IMG; src=/images/banner1.svg; alt=ASP.NET; class=img-responsive}, @{outerHTML=<img src="/images/banner2.svg"
                    alt="Visual Studio" class="img-responsive" />; tagName=IMG; src=/images/banner2.svg; alt=Visual Studio; class=img-responsive}, @{outerHTML=<img src="/images/banner3.svg" alt="Package Management"
                    class="img-responsive" />; tagName=IMG; src=/images/banner3.svg; alt=Package Management; class=img-responsive}, @{outerHTML=<img src="/images/banner4.svg" alt="Microsoft Azure" class="img-responsive" />;
                    tagName=IMG; src=/images/banner4.svg; alt=Microsoft Azure; class=img-responsive}}
InputFields       : {}
Links             : {@{outerHTML=<a class="navbar-brand" href="/">aspnetapp</a>; tagName=A; class=navbar-brand; href=/}, @{outerHTML=<a href="/">Home</a>; tagName=A; href=/}, @{outerHTML=<a href="/Home/About">About</a>;
                    tagName=A; href=/Home/About}, @{outerHTML=<a href="/Home/Contact">Contact</a>; tagName=A; href=/Home/Contact}...}
ParsedHtml        :
RawContentLength  : 8838
RelationLink      : {}

Ping aspnetapp service: This fails (expected), but shows that the hostname is resolving to the service's virtual ip (10.0.0.2)

PS C:\> ping aspnetapp

Pinging aspnetapp [10.0.0.2] with 32 bytes of data:
Request timed out.

Test web request to the aspnetapp service: Fails

PS C:\> Invoke-Webrequest aspnetapp
Invoke-Webrequest : The operation timed out
At line:1 char:1
+ Invoke-Webrequest aspnetapp
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (Method: GET, Re...rShell/6.0.0
}:HttpRequestMessage) [Invoke-WebRequest], HttpRequestException
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand

Test web request to the aspnetapp service virtual ip: Fails

PS C:\> Invoke-Webrequest 10.0.0.2
Invoke-Webrequest : The operation timed out
At line:1 char:1
+ Invoke-Webrequest 10.0.0.2
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (Method: GET, Re...rShell/6.0.0
}:HttpRequestMessage) [Invoke-WebRequest], HttpRequestException
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand

Additional Information

> docker network inspect testoverlay
[
    {
        "Name": "testoverlay",
        "Id": "81t9kimrm538egpptf2kh6fvd",
        "Created": "2017-10-17T09:13:57.4683552-04:00",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.0.0/24",
                    "Gateway": "10.0.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "08bfa5d99e774a63187fa2d740fc63097bf2ae29d2fcfd5c88cd5329d038a267": {
                "Name": "epic_goodall",
                "EndpointID": "e215a74a4fdcfe5bec153667ef2a84e3aa0e761e3594c4c0d13f0b936a2395af",
                "MacAddress": "00:15:5d:ae:29:a6",
                "IPv4Address": "10.0.0.4/24",
                "IPv6Address": ""
            },
            "274166b0d3b731108f230af7b76b9e08ed807824d588f8579b30111d7f186231": {
                "Name": "aspnetapp.1.k6qztl2jvn61jvwmq0a6p9hpc",
                "EndpointID": "b802e2a59f846284be02026ee27244dfb4d16a9d515895cbea72a249cb3931d4",
                "MacAddress": "00:15:5d:ae:20:23",
                "IPv4Address": "10.0.0.3/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4097",
            "com.docker.network.windowsshim.hnsid": "3ab13cfd-a7e3-4666-8a50-61c2dc495e0a"
        },
        "Labels": {},
        "Peers": [
            {
                "Name": "ac-devcontnr-01-d3bc48144395",
                "IP": "192.168.5.207"
            }
        ]
    }
]

> docker service inspect aspnetapp
[
    {
        "ID": "tkbxpyu4bueexw5dx87mlpogo",
        "Version": {
            "Index": 54537
        },
        "CreatedAt": "2017-10-17T15:36:03.653849Z",
        "UpdatedAt": "2017-10-17T15:36:03.7555714Z",
        "Spec": {
            "Name": "aspnetapp",
            "Labels": {},
            "TaskTemplate": {
                "ContainerSpec": {
                    "Image": "aspnetapp:latest",
                    "StopGracePeriod": 10000000000,
                    "DNSConfig": {}
                },
                "Resources": {
                    "Limits": {},
                    "Reservations": {}
                },
                "RestartPolicy": {
                    "Condition": "any",
                    "Delay": 5000000000,
                    "MaxAttempts": 0
                },
                "Placement": {
                    "Constraints": [
                        "node.hostname == ac-devcontnr-01"
                    ]
                },
                "Networks": [
                    {
                        "Target": "81t9kimrm538egpptf2kh6fvd"
                    }
                ],
                "ForceUpdate": 0,
                "Runtime": "container"
            },
            "Mode": {
                "Replicated": {
                    "Replicas": 1
                }
            },
            "UpdateConfig": {
                "Parallelism": 1,
                "FailureAction": "pause",
                "Monitor": 5000000000,
                "MaxFailureRatio": 0,
                "Order": "stop-first"
            },
            "RollbackConfig": {
                "Parallelism": 1,
                "FailureAction": "pause",
                "Monitor": 5000000000,
                "MaxFailureRatio": 0,
                "Order": "stop-first"
            },
            "EndpointSpec": {
                "Mode": "vip",
                "Ports": [
                    {
                        "Protocol": "tcp",
                        "TargetPort": 80,
                        "PublishedPort": 80,
                        "PublishMode": "host"
                    }
                ]
            }
        },
        "Endpoint": {
            "Spec": {
                "Mode": "vip",
                "Ports": [
                    {
                        "Protocol": "tcp",
                        "TargetPort": 80,
                        "PublishedPort": 80,
                        "PublishMode": "host"
                    }
                ]
            },
            "Ports": [
                {
                    "Protocol": "tcp",
                    "TargetPort": 80,
                    "PublishedPort": 80,
                    "PublishMode": "host"
                }
            ],
            "VirtualIPs": [
                {
                    "NetworkID": "81t9kimrm538egpptf2kh6fvd",
                    "Addr": "10.0.0.2/24"
                }
            ]
        }
    }
]

This works using docker-compose instead of docker service create

docker-compose.yml:

version: '3.2'

networks:
    testoverlay:
        external: true

services:

  aspnetapp:
    image: aspnetapp
    deploy:
       placement:
        constraints:
          - node.hostname == ac-devcontnr-01
    ports:
      - target: 80
        published: 80
        mode: host
    networks:
      - testoverlay

Test http request from powershell container on same overlay network:

PS C:\> Invoke-Webrequest aspnetapp

StatusCode        : 200
StatusDescription : OK
Content           : <!DOCTYPE html>
                    <html>
                    <head>
                        <meta charset="utf-8" />
                        <meta name="viewport" content="width=device-width, initial-scale=1.0" />
                        <title>Home Page - aspnetapp</title>

                       ...
RawContent        : HTTP/1.1 200 OK
                    Date: Tue, 17 Oct 2017 15:54:41 GMT
                    Transfer-Encoding: chunked
                    Server: Kestrel
                    Content-Type: text/html; charset=utf-8

                    <!DOCTYPE html>
                    <html>
                    <head>
                        <meta charset="utf-8" ...
Forms             :
Headers           : {[Date, System.String[]], [Transfer-Encoding, System.String[]], [Server, System.String[]], [Content-Type, System.String[]]}
Images            : {@{outerHTML=<img src="/images/banner1.svg" alt="ASP.NET" class="img-responsive" />; tagName=IMG; src=/images/banner1.svg; alt=ASP.NET; class=img-responsive}, @{outerHTML=<img src="/images/banner2.svg"
                    alt="Visual Studio" class="img-responsive" />; tagName=IMG; src=/images/banner2.svg; alt=Visual Studio; class=img-responsive}, @{outerHTML=<img src="/images/banner3.svg" alt="Package Management"
                    class="img-responsive" />; tagName=IMG; src=/images/banner3.svg; alt=Package Management; class=img-responsive}, @{outerHTML=<img src="/images/banner4.svg" alt="Microsoft Azure" class="img-responsive" />;
                    tagName=IMG; src=/images/banner4.svg; alt=Microsoft Azure; class=img-responsive}}
InputFields       : {}
Links             : {@{outerHTML=<a class="navbar-brand" href="/">aspnetapp</a>; tagName=A; class=navbar-brand; href=/}, @{outerHTML=<a href="/">Home</a>; tagName=A; href=/}, @{outerHTML=<a href="/Home/About">About</a>;
                    tagName=A; href=/Home/About}, @{outerHTML=<a href="/Home/Contact">Contact</a>; tagName=A; href=/Home/Contact}...}
ParsedHtml        :
RawContentLength  : 8838
RelationLink      : {}

StetsonG commented 6 years ago

So I have found a few references to the fact that the default VIP routing is not supported on Windows Containers unless you are using the newly released Windows 10/Server 1709 and a preview version of Docker EE:

These networking improvements also unlock VIP-based service discovery when using overlay networks so that Windows users are not limited to DNS Round Robin.

https://blog.docker.com/2017/09/docker-windows-server-1709/

So with that knowledge I set my services to use DNS Round Robin by setting the endpoint_mode in the docker-compose file:

deploy:
     endpoint_mode: dnsrr

And then communication between services works as expected.

I think then that this is mainly a documentation issue, as it was not very clear when looking at Microsoft's or Docker's documentation that this was a limitation of Windows Containers.

It would also make sense for docker to default to DNS Round Robin on platforms that do not support VIP networking.

docker-robott commented 6 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale comment. Stale issues will be closed after an additional 30d of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows. /lifecycle stale

masaeedu commented 4 years ago

/remove-lifecycle stale

masaeedu commented 4 years ago

@StetsonG The blog post indicates that virtual IP is now supported, however this still doesn't work on Windows Server 2019 a few years later.

docker-robott commented 4 years ago

Closed issues are locked after 30 days of inactivity. This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows. /lifecycle locked

docker / for-win