Can't publish a port for a service on v4.24+

GuillaumeRossolini commented 11 months ago

Description

TL;DR: see @mcgodfrey's comment below for a more succinct description and reproduce steps: https://github.com/docker/for-win/issues/13721#issuecomment-1806636232

The latest v4.24.0 seems to behave differently network-wise from v4.23.0 and all earlier versions, as far as published ports on services inside a stack are concerned.

I'm not sure where the issue is precisely, so I'll try to summarize my observations. I'm opening this mostly to see it that has been identified yet, and I'll share a precise use case if needed.

Reverting back to Docker Desktop v4.23.0 fixes this for me.

Reproduce

Spin up a service inside a stack, and tell the swarm to bind the port 80 in host/vip mode. Something along the lines of:

version: "3.8"
networks:
  default:
    driver: overlay
services:
  localdevhostname:
    image: ChooseYourHttpServer
    deploy:
      placement:
        constraints: []
      replicas: 1
    networks:
      default: null
    ports:
      - "80:80"

In my case, after the service is working and I can inspect it with:

docker service inspect MyFullServiceName | jq -r ".[0].Spec"

Which has the same output for v4.23.0 to v4.24.0:

(clipped)
  "Mode": {
    "Replicated": {
      "Replicas": 1
    }
  },
  "UpdateConfig": {
    "Parallelism": 1,
    "FailureAction": "pause",
    "Monitor": 5000000000,
    "MaxFailureRatio": 0,
    "Order": "stop-first"
  },
  "RollbackConfig": {
    "Parallelism": 1,
    "FailureAction": "pause",
    "Monitor": 5000000000,
    "MaxFailureRatio": 0,
    "Order": "stop-first"
  },
  "EndpointSpec": {
    "Mode": "vip",
    "Ports": [
      {
        "Protocol": "tcp",
        "TargetPort": 80,
        "PublishedPort": 80,
        "PublishMode": "host"
      }
    ]
  }
}

I can see that the service is healthy and the port is bound

docker service ls | grep MyFullServiceName | rev | cut -d ' ' -f1 | rev
*:80->80/tcp

If I then docker exec... inside a container (from this service), I can then use curl to browse its own hostname or any alias that was defined on the service. The service itself isn't an issue, it is healthy.

However, from WSL I can't do the same:

$ curl -v -D - "http://localdevhostname" --resolve localdevhostname:80:127.0.0.1
* Added localdevhostname:80:127.0.0.1 to DNS cache
* Hostname localdevhostname was found in DNS cache
*   Trying 127.0.0.1:80...
* Connected to localdevhostname (127.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> Host: localdevhostname
> User-Agent: curl/7.74.0
> Accept: */*
>
* Empty reply from server
* Connection #0 to host localdevhostname left intact
curl: (52) Empty reply from server

NB: If I use the short YAML syntax as described above to bind ports, Docker Desktop Dashboard doesn't see it as open but docker service ls does. If I use the detailed YAML syntax (see below), it's the other way around. In both cases, v4.23.0 allows me to browse from the host. In neither case v4.24.0 lets me do that.

ports:
  - published: 80
    target: 80
    protocol: tcp
    mode: host

I then uninstalled Docker Desktop v4.24.0, reinstalled v4.23.0, started the app again and my local stack works again, no changes required.

Expected behavior

if docker service ls says a port is bound, the host should be able to reach it
if Docker Dashboard says a port is open, the host should be able to reach it

docker version

(this after downgrading Docker Desktop from v4.24.0 to v4.23.0)

Client:
 Version:           20.10.5+dfsg1
 API version:       1.41
 Go version:        go1.15.15
 Git commit:        55c4c88
 Built:             Mon May 30 18:34:49 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Desktop 4.23.0 (120376)
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:32:16 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.22
  GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

(this after downgrading Docker Desktop from v4.24.0 to v4.23.0)

Client:
 Context:    default
 Debug Mode: false
 Plugins:
WARNING: Plugin "/usr/local/lib/docker/cli-plugins/docker-buildx" is not valid: failed to fetch metadata: fork/exec /usr/local/lib/docker/cli-plugins/docker-buildx: no such file or directory
WARNING: Plugin "/usr/local/lib/docker/cli-plugins/docker-compose" is not valid: failed to fetch metadata: fork/exec /usr/local/lib/docker/cli-plugins/docker-compose: no such file or directory
WARNING: Plugin "/usr/local/lib/docker/cli-plugins/docker-dev" is not valid: failed to fetch metadata: fork/exec /usr/local/lib/docker/cli-plugins/docker-dev: no such file or directory
WARNING: Plugin "/usr/local/lib/docker/cli-plugins/docker-extension" is not valid: failed to fetch metadata: fork/exec /usr/local/lib/docker/cli-plugins/docker-extension: no such file or directory
WARNING: Plugin "/usr/local/lib/docker/cli-plugins/docker-init" is not valid: failed to fetch metadata: fork/exec /usr/local/lib/docker/cli-plugins/docker-init: no such file or directory
WARNING: Plugin "/usr/local/lib/docker/cli-plugins/docker-sbom" is not valid: failed to fetch metadata: fork/exec /usr/local/lib/docker/cli-plugins/docker-sbom: no such file or directory
WARNING: Plugin "/usr/local/lib/docker/cli-plugins/docker-scan" is not valid: failed to fetch metadata: fork/exec /usr/local/lib/docker/cli-plugins/docker-scan: no such file or directory
WARNING: Plugin "/usr/local/lib/docker/cli-plugins/docker-scout" is not valid: failed to fetch metadata: fork/exec /usr/local/lib/docker/cli-plugins/docker-scout: no such file or directory

Server:
 Containers: 96
  Running: 94
  Paused: 0
  Stopped: 2
 Images: 71
 Server Version: 24.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: x1pd4tpnhspg7e4a0swq95hzu
  Is Manager: true
  ClusterID: x3en8cmao9ted3oh2vao2qy99
  Managers: 1
  Nodes: 1
  Default Address Pool: 10.0.0.0/8
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 192.168.65.4
  Manager Addresses:
   192.168.65.4:2377
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  seccomp
  WARNING: You're not using the default seccomp profile
   Profile: unconfined
 Kernel Version: 5.15.90.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 7.761GiB
 Name: docker-desktop
 ID: 027abfb4-3aaa-4f31-b074-e23729d32552
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: daemon is not using the default seccomp profile

Diagnostics ID

97770725-A6F9-4820-A37B-9641A5FD1CD9/20231004155030

Additional Info

No response

flaviosiotto commented 11 months ago

same issue also in version 4.23.1

GuillaumeRossolini commented 10 months ago

Upgraded from v4.23.0 to v4.25.0, same issue: nothing gets routed to my containers, they aren't logging any hits. Uninstalled Docker Desktop, rebooted the PC and installed v4.25.0 fresh: same issue. Unisntalled v4.25.0, installed v4.24.2 fresh: same issue. Unisntalled v4.24.2, installed v4.23.0 fresh: FIXED.

Windows 10 is fully up to date (22H2 build 19045).

Re-stating the issue with v4.24.0+: I am using docker stack/swarm services. Anything I look at looks ok, be it docker ps or Docker Desktop Dashboard. The ports I expect are bound, but the services don't get the inbound trafic and they don't respond.

I also tried docker stack rm and re-creating my entire stack, networks and all, but no luck.

GuillaumeRossolini commented 10 months ago

@flaviosiotto I can't find that version, did you mean 4.24.1 by any chance?

mcgodfrey commented 10 months ago

I'm having the same problem with v4.24.x and v4.25.0.

Simple steps to reproduce:

V4.23.0 - working

install docker desktop v4.23.0
docker swarm init
docker service create --name proxy --publish 4000:80 nginx
navigate to http://localhost:4000 in a browser and see default nginx page

V4.25.0 - not working

install docker desktop v4.25.0
docker swarm init
docker service create --name proxy --publish 4000:80 nginx
navigate to http://localhost:4000 in a browser - time out, no response

Extra info

I also ran a docker service inspect proxy - identical between v4.23.0 an 4.25.0 (except for ID and timestamps). In both cases, the endpoint spec says that the port is mapped, but it is inaccessible from the host in V4.25.0.

docker service inspect proxy in docker desktop V4.25.0:

[
    {
        "ID": "75ywa8vbx34yj85r6dzshtku7",
        "Version": {
            "Index": 48
        },
        "CreatedAt": "2023-11-11T02:12:09.354374035Z",
        "UpdatedAt": "2023-11-11T02:12:09.357417547Z",
        "Spec": {
            "Name": "proxy",
            "Labels": {},
            "TaskTemplate": {
                "ContainerSpec": {
                    "Image": "nginx:latest@sha256:86e53c4c16a6a276b204b0fd3a8143d86547c967dc8258b3d47c3a21bb68d3c6",
                    "Init": false,
                    "StopGracePeriod": 10000000000,
                    "DNSConfig": {},
                    "Isolation": "default"
                },
                "Resources": {
                    "Limits": {},
                    "Reservations": {}
                },
                "RestartPolicy": {
                    "Condition": "any",
                    "Delay": 5000000000,
                    "MaxAttempts": 0
                },
                "Placement": {
                    "Platforms": [
                        {
                            "Architecture": "amd64",
                            "OS": "linux"
                        },
                        {
                            "OS": "linux"
                        },
                        {
                            "OS": "linux"
                        },
                        {
                            "Architecture": "arm64",
                            "OS": "linux"
                        },
                        {
                            "Architecture": "386",
                            "OS": "linux"
                        },
                        {
                            "Architecture": "mips64le",
                            "OS": "linux"
                        },
                        {
                            "Architecture": "ppc64le",
                            "OS": "linux"
                        },
                        {
                            "Architecture": "s390x",
                            "OS": "linux"
                        }
                    ]
                },
                "ForceUpdate": 0,
                "Runtime": "container"
            },
            "Mode": {
                "Replicated": {
                    "Replicas": 1
                }
            },
            "UpdateConfig": {
                "Parallelism": 1,
                "FailureAction": "pause",
                "Monitor": 5000000000,
                "MaxFailureRatio": 0,
                "Order": "stop-first"
            },
            "RollbackConfig": {
                "Parallelism": 1,
                "FailureAction": "pause",
                "Monitor": 5000000000,
                "MaxFailureRatio": 0,
                "Order": "stop-first"
            },
            "EndpointSpec": {
                "Mode": "vip",
                "Ports": [
                    {
                        "Protocol": "tcp",
                        "TargetPort": 80,
                        "PublishedPort": 4000,
                        "PublishMode": "ingress"
                    }
                ]
            }
        },
        "Endpoint": {
            "Spec": {
                "Mode": "vip",
                "Ports": [
                    {
                        "Protocol": "tcp",
                        "TargetPort": 80,
                        "PublishedPort": 4000,
                        "PublishMode": "ingress"
                    }
                ]
            },
            "Ports": [
                {
                    "Protocol": "tcp",
                    "TargetPort": 80,
                    "PublishedPort": 4000,
                    "PublishMode": "ingress"
                }
            ],
            "VirtualIPs": [
                {
                    "NetworkID": "mopovoesevldk105kayv1khkt",
                    "Addr": "10.0.0.3/24"
                }
            ]
        }
    }
]

Likewise, except for IDs and timestamps, the results of docker network inspect ingress are identical between the 2 versions too.

docker network inspect ingress in docker desktop V4.25.0:

[
    {
        "Name": "ingress",
        "Id": "mopovoesevldk105kayv1khkt",
        "Created": "2023-11-11T02:11:29.374991305Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.0.0/24",
                    "Gateway": "10.0.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": true,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "a870baa0e93c9961f73891df75e315153a5d8b56ac0773d62668e8711a1dfa46": {
                "Name": "proxy.1.wme3fv3wxnpk84cbqgbxd6pwl",
                "EndpointID": "2337dcc0d9c07d893c318d7ded33e230d917c00ab897def84437b0626635a448",
                "MacAddress": "02:42:0a:00:00:04",
                "IPv4Address": "10.0.0.4/24",
                "IPv6Address": ""
            },
            "ingress-sbox": {
                "Name": "ingress-endpoint",
                "EndpointID": "61c7e7723c472c1ce8f284f7161cbd790d569cd720307ce2ed4f257ec66810f2",
                "MacAddress": "02:42:0a:00:00:02",
                "IPv4Address": "10.0.0.2/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4096"
        },
        "Labels": {},
        "Peers": [
            {
                "Name": "32c30e955dc5",
                "IP": "192.168.65.4"
            }
        ]
    }
]

johnmanko commented 9 months ago

This is also a problem with Docker Desktop for Linux.

textgroove-steven commented 8 months ago

I am running into this same issue. It seems docker swarm is broken networking wise above 4.23.

Paul2708 commented 3 weeks ago

I faced the same issue months ago as well. Thus, I did not upgrade. However, I just re-installed Docker Desktop, and using the latest version (currently 4.33.1 (161083)) seems to fix the issue; services can be reached after publishing them.

docker / for-win