Docker 20.10.01 doesn't release port after stopped the container

BSthun commented 3 years ago

[x] This is a bug report
[ ] This is a feature request
[ ] I searched existing issues before opening this one

Expected behavior

After stopped the container that binded the host port. The binded port should be released and any containers (including itself) are able to bind that port afterward.

Actual behavior

I started a container that bind (expose) the port on host. At first time when started the container, it's able to start successfully, but after I've stopped that container (or even stopped and run docker system prune -a) and run it again, it's says 0.0.0.0:443: bind: address already in use and 2607:f130:0:f8::4290:6146:443 failed: port is already allocated which means binded port since first time of starting container hasn't been release yet.

Workaround

Stop and remove the container then restart Docker (systemctl restart docker)
Use older version of Docker instead (it works on 5:19.03.14~3-0~debian-buster)

Steps to reproduce the behavior

I'm running Docker on Debian 10 buster (fresh install) using docker-ce (5:20.10.1~3-0~debian-buster), docker-ce-cli (5:20.10.1~3-0~debian-buster) with Docker Compose.

My Docker Compose file

version: '3.5'

services:
    web:
        container_name: frontpxy
        image: nginx:alpine
        hostname: frontpxy
        domainname: server1.bsthun.com
        networks:
            default:
                ipv4_address: 10.5.1.2
        ports:
            - "2607:f130:0:f8::4290:6146:443:443"
        volumes:
            - ./sites:/etc/nginx/conf.d
            - ./ssl:/etc/nginx/ssl
        restart: always
        mem_limit: 512m
        cpus: 2
        logging:
            driver: none
networks:
    default:
        external:
            name: br

First time I started the container using, it's works correctly. After I've stop them and bring it up again (with no any modification to docker and docker compose file), it says

Starting cortex ... error

ERROR: for cortex  Cannot start service web: driver failed programming external connectivity on endpoint cortex (e13755b817f24042a7a158c2ab7486f18a3b56875cc48be6081b9923ed5061a2): Bind for 2607:f130:0:f8::4290:6146:443 failed: port is already allocated

ERROR: for web  Cannot start service web: driver failed programming external connectivity on endpoint cortex (e13755b817f24042a7a158c2ab7486f18a3b56875cc48be6081b9923ed5061a2): Bind for 2607:f130:0:f8::4290:6146:443 failed: port is already allocated
ERROR: Encountered errors while bringing up the project.

By these same steps, but with Docker 5:19.03.14~3-0~debian-buster, it's works correctly with no any issue.

Output of docker version:

Client: Docker Engine - Community
 Version:           20.10.1
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        831ebea
 Built:             Tue Dec 15 04:34:48 2020
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.1
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       f001486
  Built:            Tue Dec 15 04:32:45 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.0-docker)

Server:
 Containers: 2
  Running: 1
  Paused: 0
  Stopped: 1
 Images: 2
 Server Version: 20.10.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.19.0-6-amd64
 Operating System: Debian GNU/Linux 10 (buster)
 OSType: linux
 Architecture: x86_64
 CPUs: 3
 Total Memory: 3.362GiB
 Name: bsthun-server1
 ID: UM3F:2QVI:P3N5:F3ZS:JZYV:QWSW:OMGH:BB6T:DWRV:4MQ6:GSCH:5VPB
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.)

arkodg commented 3 years ago

hey @BSthun do you see the same issue with Docker CE 20.10.2

BSthun commented 3 years ago

Wow, the issue have resolved in Docker CE 20.10.2. Thank you so much.

BSthun commented 3 years ago

Hello again. From that issue, I found another issue that I think it's related to. On Docker CE 20.10.2, yes it released the port, but it's also doesn't bind the host port at all.

From the same compose file as above (https://github.com/docker/for-linux/issues/1180#issue-778577235), I after I run docker-compose up, the container created, but unfortunately that it's doesn't bind the host port even set the compose file to expose that port. In docker ps, it's show only the exposed port of the container without binding to external IP or 0.0.0.0. Like this: And the same, I use the same config to run on Docker CE 19.03.14, with same host and environment (fresh reinstall debian 10) it works fine. I'm not sure this is issue of Docker CE or Docker Compose. but changing version of Docker works for me.

Mehrdad-Dadkhah commented 2 years ago

Same problem at Docker version 20.10.9, build c2ea9bc90b

zldrobit commented 2 years ago

Same problem at Docker version 20.10.8, build 3967b7d with kenel 5.11.0-40-generic on Ubuntu 20.04.3 LTS.

giorgionicoli commented 2 years ago

Same problem with Docker version 20.10.13, build a224086 on MacOS Monterey with M1

khoa-nguyendang commented 2 years ago

Same problem version 20.10.13 on Windows, WSL2

GCSBOSS commented 2 years ago

It seems to me I'm having the same issue on Docker Desktop for Windows.

prabhu-vinayaka commented 2 years ago

The issue have resolved in Docker CE 20.10.2. Could you backport it to any 19.x version please? Thank you.

enri-kapaj commented 8 months ago

Im faceing the same issue on my mac m2 pro When i first run the docker-compose up the container works fine but when it stops i get that the port is already in use : Docker version :24.0.7 Error :

de:net:1876
cmdb-integration-1  |       const error = new UVExceptionWithHostPort(rval, 'listen', address, port);
cmdb-integration-1  |                     ^
cmdb-integration-1  | 
cmdb-integration-1  | Error: listen EADDRINUSE: address already in use /tmp/tsx-0/33.pipe
cmdb-integration-1  |     at Server.setupListenHandle [as _listen2] (node:net:1876:21)
cmdb-integration-1  |     at listenInCluster (node:net:1941:12)
cmdb-integration-1  |     at Server.listen (node:net:2046:5)
cmdb-integration-1  |     at file:///cmdb-integration/node_modules/tsx/dist/cli.mjs:53:31317
cmdb-integration-1  |     at new Promise (<anonymous>)
cmdb-integration-1  |     at yn (file:///cmdb-integration/node_modules/tsx/dist/cli.mjs:53:31295)
cmdb-integration-1  |     at async file:///cmdb-integration/node_modules/tsx/dist/cli.mjs:53:32265 {
cmdb-integration-1  |   code: 'EADDRINUSE',
cmdb-integration-1  |   errno: -98,
cmdb-integration-1  |   syscall: 'listen',
cmdb-integration-1  |   address: '/tmp/tsx-0/33.pipe',
cmdb-integration-1  |   port: -1
cmdb-integration-1  | }

BSthun commented 8 months ago

Seems like you're binding to unix socket. Could you try remove the container (docker compose down) and start it again?

enri-kapaj commented 8 months ago

yep i did that and still have the same issue(it runs the first time when i do compose up again but not the second time )

ehoang40 commented 6 months ago

I'm encountering this issue as well. The container works fine the first time running docker compose up, but if I stop the container then run compose up again, it errors with Error: listen EADDRINUSE: address already in use. If I destroy the containers with docker compose down and then run compose up, the new containers work fine. But it would be nice to reuse the same containers instead of having to destroy/create new ones each time. Machine: Mac M1 Pro, macOS Monterey Docker version: 23.0.0, build e92dd87c32

ZantaShock commented 1 month ago

I ran into the same Problem. I am using Ubuntu 22.04.4. I have multiple machines but only the one running Docker version 27.0.1, build 7fafd33 seems to have this problem. The machines with Docker version 26.x.x are all running as excepted.

akerouanton commented 1 month ago

@ZantaShock Could you please try to:

Stop any running container and kill docker-proxy processes that aren't cleaned up ;
Then, enable debug logs (see here: https://docs.docker.com/engine/daemon/logs/#enable-debugging) ;
Start your container, do a docker inspect of that container, and then stop it and restart it again to reproduce the issue ;
Paste here both the docker inspect output and all the logs generated by the daemon during the previous step.

Also, could you check that the port is really bound with ss -lpn | grep <YOURPORT>.

ZantaShock commented 1 month ago

@akerouanton Killing docker-proxy processes worked. It the only workaround I had. I managed to resolve the problem by upgrading docker to the newest version.

akerouanton commented 1 month ago

If you (or anyone else) ever run into this again, please try to reproduce the issue and send us everything mentioned in this comment: https://github.com/docker/for-linux/issues/1180#issuecomment-2301323708. Otherwise we won't be able to solve it.

ZantaShock commented 1 month ago

@ZantaShock Could you please try to:

* Stop any running container and kill `docker-proxy` processes that aren't cleaned up ;

* Then, enable debug logs (see here: https://docs.docker.com/engine/daemon/logs/#enable-debugging) ;

* Start your container, do a `docker inspect` of that container, and then stop it and restart it again to reproduce the issue ;

* Paste here both the `docker inspect` output and all the logs generated by the daemon during the previous step.

Also, could you check that the port is really bound with ss -lpn | grep <YOURPORT>.

I encountered this Issue again after the Ubuntu-Server crashed. The problem disappered after killing the docker-proxy. Using docker stop seems to clean up correctly. Also the port is really bound even after a restart of the Server.

ZantaShock commented 1 month ago

I think I found the Issue. I noticed a stranger behavior hinting that multiple instances of a application where running on the same server. After stopping the only visible container and the application was still available I found this thread:

https://forums.docker.com/t/docker-ps-a-doesnt-show-running-containers/121046/5

Docker was in fact installed with snap and apt.

delanym commented 3 weeks ago

@akerouanton I stopped all containers and then I see in the processes /usr/bin/rootlesskit-docker-proxy -proto tcp -host-ip 10.1.1.222 -host-port 80 -container-ip 172.23.0.3 -container-port 80 So I killed that. Then I ran docker compose caddy for

services:                                                                                                                      
  caddy:    
    image: caddy:alpine       
    depends_on:                 
      - gitlab                  
    restart: unless-stopped
    ports:                                                                                                                     
      - "${GITLAB_HOST}:80:80"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
...

resulting in

[+] Running 2/0
 ✔ Container gitlab-gitlab-1  Running                                                                                                                                                                                                                    0.0s 
 ✔ Container gitlab-caddy-1   Created                                                                                                                                                                                                                    0.0s 
Attaching to caddy-1
Gracefully stopping... (press Ctrl+C again to force)
Error response from daemon: driver failed programming external connectivity on endpoint gitlab-caddy-1 (effe7a42689db5235beb8036c1d2c42bd0322ec5cad04fbffa1efcc386b44e40): failed to bind port 10.1.1.222:80/tcp: Error starting userland proxy: error while calling PortManager.AddPort(): conflict with ID 6

and lo and behold rootlesskit-docker-proxy is back listening on port 10.1.1.222:80

I setup the daemon on this machine to run in rootless mode, and I also apply this capability https://docs.docker.com/engine/security/rootless/#exposing-privileged-ports I can confirm via sudo setcap -v cap_net_bind_service=ep $(which rootlesskit) that it has the capability /usr/bin/rootlesskit: OK

Docker version 27.2.0, build 3ab4256 rootlesskit version 2.0.2 Ubuntu 24.04

delanym commented 2 weeks ago

Should I make another issue? My container doesn't even start

docker / for-linux