search

docker / cli

The Docker CLI
Apache License 2.0
4.98k stars 1.94k forks source link

Cannot start service xyz: Ports are not available: listen tcp 0.0.0.0:6904: bind: address already in use #3119

Open jamesmortensen opened 3 years ago

jamesmortensen commented 3 years ago

BUG REPORT INFORMATION

Description

I use docker-compose to launch a Selenium hub with 3 identical Chrome containers. I modified the configuration so that I could use the --scale argument to scale the number of Chrome containers up or down depending on the need. I launch the Chrome containers from selenium/node-chrome images, which have a VNC server running on port 5900. I defined a docker-compose.yml with the hub and a Chrome image.

When running the command to start the containers, and then shut them down, if I then try to start them again, sometimes one of the VNC ports exposed on the host side are blocked by Docker. When running docker ps there are no containers running, yet the port is still blocked.

Here is the docker-compose:

version: "3"
services:
  chrome:
    image: selenium/node-chrome:4.0.0-beta-3-prerelease-20210422
    volumes:
      - /dev/shm:/dev/shm
    depends_on:
      - selenium-hub
    environment:
      - SE_EVENT_BUS_HOST=selenium-hub
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443
    ports:
      - "6900-6909:5900"

  selenium-hub:
    image: selenium/hub:4.0.0-beta-3-prerelease-20210422
    container_name: selenium-hub
    ports:
      - "4442:4442"
      - "4443:4443"
      - "4444:4444"

Steps to reproduce the issue:

  1. Run docker-compose -f docker-compose.yml up -d --scale chrome=3
  2. The 3 chrome containers are now accessible via VNC on ports 6900, 6901, and 6902, and you can verify with VNCViewer or RealVNC.
  3. Stop the containers: docker-compose -f docker-compose.yml down
  4. Use docker ps to verify the containers have shutdown. Use docker ps -a to verify the containers have been removed.
  5. Again, run docker-compose -f docker-compose.yml up -d --scale chrome=3 This time, one of the containers may not be started due to one of the 6900-6909 ports being blocked by Docker.

Describe the results you received:

After bringing the containers down, removing them, and then restarting them, one of the ports is blocked. In this example, port 6903 is blocked.

$ docker-compose -f .wdio-ci-configs/parallel-tests-on-selenium-grid/docker-compose-hub-parallel.yml up -d --scale chrome=3
Docker Compose is now in the Docker CLI, try `docker compose up`

Creating network "parallel-tests-on-selenium-grid_default" with the default driver
Creating selenium-hub ... done
WARNING: The "chrome" service specifies a port on the host. If multiple containers for this service are created on a single host, the port will clash.
Creating parallel-tests-on-selenium-grid_chrome_1 ... error
Creating parallel-tests-on-selenium-grid_chrome_2 ... done
Creating parallel-tests-on-selenium-grid_chrome_3 ... done

ERROR: for parallel-tests-on-selenium-grid_chrome_1  Cannot start service chrome: Ports are not available: listen tcp 0.0.0.0:6903: bind: address already in use

ERROR: for chrome  Cannot start service chrome: Ports are not available: listen tcp 0.0.0.0:6903: bind: address already in use
ERROR: Encountered errors while bringing up the project.

Also, the ports were not selected in order. Below we see ports 6900 and 6909 were bound successfully while port 6903 was marked as being already in use, even though it wasn't.

$ docker ps
CONTAINER ID   IMAGE                                                   COMMAND                  CREATED              STATUS              PORTS                                                           NAMES
348b9653600c   selenium/node-chrome:4.0.0-beta-3-prerelease-20210422   "/opt/bin/entry_poin…"   About a minute ago   Up About a minute   0.0.0.0:6900->5900/tcp, :::6900->5900/tcp                       parallel-tests-on-selenium-grid_chrome_2
0aeabe873320   selenium/node-chrome:4.0.0-beta-3-prerelease-20210422   "/opt/bin/entry_poin…"   About a minute ago   Up About a minute   0.0.0.0:6909->5900/tcp, :::6909->5900/tcp                       parallel-tests-on-selenium-grid_chrome_3
5f3baf558790   selenium/hub:4.0.0-beta-3-prerelease-20210422           "/opt/bin/entry_poin…"   About a minute ago   Up About a minute   0.0.0.0:4442-4444->4442-4444/tcp, :::4442-4444->4442-4444/tcp   selenium-hub

Describe the results you expected:

I was expecting to be able to scale the containers up to the number of available ports in the range 6900-6909 and have Docker free up the ports when bringing the containers down. However, some of the ports remain blocked.

Sometimes, like in this example, everything works as expected:

$ docker-compose -f .wdio-ci-configs/parallel-tests-on-selenium-grid/docker-compose-hub-parallel.yml up -d --scale chrome=3
Docker Compose is now in the Docker CLI, try `docker compose up`

Creating network "parallel-tests-on-selenium-grid_default" with the default driver
Creating selenium-hub ... done
WARNING: The "chrome" service specifies a port on the host. If multiple containers for this service are created on a single host, the port will clash.
Creating parallel-tests-on-selenium-grid_chrome_1 ... done
Creating parallel-tests-on-selenium-grid_chrome_2 ... done
Creating parallel-tests-on-selenium-grid_chrome_3 ... done

Additional information you deem important

Before attempting to scale the containers, we had two docker-compose files, one with 3 chrome containers each labelled chrome0, chrome1, chrome2 and then the other one with chrome0, chrome1, chrome2 and chrome3. When bringing those containers up and down, none of the ports became blocked. It was only when we attempted to use a single configuration for the chrome containers that we experience blocked VNC ports. Here is the original configuration of one of the files, as an example:

version: "3"
services:
  chrome0:
    image: selenium/node-chrome:4.0.0-beta-3-prerelease-20210422
    volumes:
      - /dev/shm:/dev/shm
    depends_on:
      - selenium-hub
    environment:
      - SE_EVENT_BUS_HOST=selenium-hub
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443
    ports:
      - "6900:5900"

  chrome1:
    image: selenium/node-chrome:4.0.0-beta-3-prerelease-20210422
    volumes:
      - /dev/shm:/dev/shm
    depends_on:
      - selenium-hub
    environment:
      - SE_EVENT_BUS_HOST=selenium-hub
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443
    ports:
      - "6901:5900"

  chrome2:
    image: selenium/node-chrome:4.0.0-beta-3-prerelease-20210422
    volumes:
      - /dev/shm:/dev/shm
    depends_on:
      - selenium-hub
    environment:
      - SE_EVENT_BUS_HOST=selenium-hub
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443
    ports:
      - "6902:5900"

  selenium-hub:
    image: selenium/hub:4.0.0-beta-3-prerelease-20210422
    container_name: selenium-hub
    ports:
      - "4442:4442"
      - "4443:4443"
      - "4444:4444"

With 3 separate, yet identical containers, aside from the VNC port, there were no conflicts.

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
  compose: Docker Compose (Docker Inc., 2.0.0-beta.1)
  scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Containers: 10
  Running: 0
  Paused: 0
  Stopped: 10
 Images: 35
 Server Version: 20.10.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.25-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 3.844GiB
 Name: docker-desktop
 ID: TSJO:KAYN:ZLE3:3LNW:WQMX:WBD4:LPVL:RTV4:Y4AB:JL27:SEK2:OGA5
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 44
  Goroutines: 49
  System Time: 2021-06-03T16:43:25.932805464Z
  EventsListeners: 4
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Output of docker version:

20.10.6 build 370c289

Additional environment details:

OS: Macbook Pro 10.15.1 Shell: Bash Docker version: 20.10.6 build 370c289

jamesmortensen commented 3 years ago

Also, I ran netstat while the containers were down. No port 6900-6909 is listed, yet we see two ports are blocked.

$ netstat -an | grep LISTEN
tcp4       0      0  127.0.0.1.54395        *.*                    LISTEN     
tcp4       0      0  127.0.0.1.49757        *.*                    LISTEN     
tcp4       0      0  127.0.0.1.49750        *.*                    LISTEN     
tcp4       0      0  127.0.0.1.49749        *.*                    LISTEN     
tcp4       0      0  *.49469                *.*                    LISTEN     
tcp6       0      0  *.61500                *.*                    LISTEN     
tcp4       0      0  *.61500                *.*                    LISTEN     
tcp4       0      0  *.88                   *.*                    LISTEN     
tcp6       0      0  *.88                   *.*                    LISTEN     
tcp4       0      0  *.445                  *.*                    LISTEN     
tcp6       0      0  *.445                  *.*                    LISTEN     

$ docker-compose -f .wdio-ci-configs/parallel-tests-on-selenium-grid/docker-compose-hub-parallel.yml up -d --scale chrome=3
Docker Compose is now in the Docker CLI, try `docker compose up`

Creating network "parallel-tests-on-selenium-grid_default" with the default driver
Creating selenium-hub ... done
WARNING: The "chrome" service specifies a port on the host. If multiple containers for this service are created on a single host, the port will clash.
Creating parallel-tests-on-selenium-grid_chrome_1 ... error
Creating parallel-tests-on-selenium-grid_chrome_2 ... done
Creating parallel-tests-on-selenium-grid_chrome_3 ... error

ERROR: for parallel-tests-on-selenium-grid_chrome_1  Cannot start service chrome: Ports are not available: listen tcp 0.0.0.0:6903: bind: address already in use

ERROR: for parallel-tests-on-selenium-grid_chrome_3  Cannot start service chrome: Ports are not available: listen tcp 0.0.0.0:6905: bind: address already in use

ERROR: for chrome  Cannot start service chrome: Ports are not available: listen tcp 0.0.0.0:6903: bind: address already in use
ERROR: Encountered errors while bringing up the project.
StefanScherer commented 3 years ago

Hi @jamesmortensen I've tried with Docker Desktop 3.5.1 on my Intel Mac and I cannot reproduce the problem.

$ docker-compose -f docker-compose.yml up -d --scale chrome=3
Creating network "cli3119_default" with the default driver
Creating selenium-hub ... done
WARNING: The "chrome" service specifies a port on the host. If multiple containers for this service are created on a single host, the port will clash.
Creating cli3119_chrome_1 ... done
Creating cli3119_chrome_2 ... done
Creating cli3119_chrome_3 ... done

then 

docker-compose down

and repeated that four times. I recommend to update Docker Desktop to the latest and try again.

mat007 commented 3 years ago

@jamesmortensen what’s the output of 

docker-compose --version

?

edit: oh never mind I just noticed you had Docker Compose is now in the Docker CLI, try docker compose up so that must be Compose V1

StefanScherer commented 3 years ago

I also tested docker compose up with Compose v2 beta 4. In this case I see an error

$ docker compose up -d --scale chrome=3
[+] Running 1/4
 ⠿ Container selenium-hub      Started                                                                                           17.6s
 ⠿ Container cli3119_chrome_1  Starting                                                                                          12.8s
 ⠿ Container cli3119_chrome_2  Starting                                                                                          12.8s
 ⠿ Container cli3119_chrome_3  Starting                                                                                          12.8s
Error response from daemon: Ports are not available: listen tcp 0.0.0.0:6900: bind: address already in use

and in the VM I see all ports listening

/ # netstat -an |grep LISTEN |grep 69
tcp        0      0 0.0.0.0:6900            0.0.0.0:*               LISTEN      
tcp        0      0 0.0.0.0:6901            0.0.0.0:*               LISTEN      
tcp        0      0 0.0.0.0:6902            0.0.0.0:*               LISTEN      
tcp        0      0 0.0.0.0:6903            0.0.0.0:*               LISTEN      
tcp        0      0 0.0.0.0:6904            0.0.0.0:*               LISTEN      
tcp        0      0 0.0.0.0:6905            0.0.0.0:*               LISTEN      
tcp        0      0 0.0.0.0:6906            0.0.0.0:*               LISTEN      
tcp        0      0 0.0.0.0:6907            0.0.0.0:*               LISTEN      
tcp        0      0 0.0.0.0:6908            0.0.0.0:*               LISTEN      
tcp        0      0 0.0.0.0:6909            0.0.0.0:*               LISTEN      
tcp        0      0 :::6900                 :::*                    LISTEN      
tcp        0      0 :::6901                 :::*                    LISTEN      
tcp        0      0 :::6902                 :::*                    LISTEN      
tcp        0      0 :::6903                 :::*                    LISTEN      
tcp        0      0 :::6904                 :::*                    LISTEN      
tcp        0      0 :::6905                 :::*                    LISTEN      
tcp        0      0 :::6906                 :::*                    LISTEN      
tcp        0      0 :::6907                 :::*                    LISTEN      
tcp        0      0 :::6908                 :::*                    LISTEN      
tcp        0      0 :::6909                 :::*                    LISTEN

Ping @ndeloof this might be a little difference between classic docker-compose and Compose v2. It seems it already allocates all 10 host ports for the first container in Compose v2.

With classic docker-compose I only saw three ports listening, but it used a different range in the - "6900-6909:5900" total range.

/ # netstat -an |grep LISTEN |grep 69
tcp        0      0 0.0.0.0:6905            0.0.0.0:*               LISTEN      
tcp        0      0 0.0.0.0:6906            0.0.0.0:*               LISTEN      
tcp        0      0 0.0.0.0:6907            0.0.0.0:*               LISTEN      
tcp        0      0 :::6905                 :::*                    LISTEN      
tcp        0      0 :::6906                 :::*                    LISTEN      
tcp        0      0 :::6907                 :::*                    LISTEN
ndeloof commented 3 years ago

duplicates https://github.com/docker/compose-cli/issues/1842

StefanScherer commented 3 years ago

Thank you @ndeloof for the link. ❤️ 🎉

@jamesmortensen Could you check your docker-compose --version ? If it shows Docker Compose version 2.0.0-beta.x then we have narrowed down the problem. In that case you could opt out of the Compose V2 in the experimental features in Docker Desktop settings for now.

kevit-dhruva-pambhar commented 2 years ago

Screenshot from 2022-09-14 11-03-28 kill the process on that port this way

jamesmortensen commented 2 years ago

@StefanScherer sorry for losing track of this issue. Here is the information you wanted:

$ docker-compose --version
docker-compose version 1.29.2, build 5becea4c
$ l /usr/local/bin/docker-compose
lrwxr-xr-x  1 root  admin  62 17 Jun  2021 /usr/local/bin/docker-compose -> /Applications/Docker.app/Contents/Resources/bin/docker-compose
$ l /usr/local/bin/docker
lrwxr-xr-x  1 root  wheel  54  5 May  2020 /usr/local/bin/docker -> /Applications/Docker.app/Contents/Resources/bin/docker

I still have the pre 4.0 Docker Desktop installed, so this should be the same docker-compose I was using around the time of the bug report. However, I can't be 100% sure, since I did switch laptops since then, and since I now run the Docker engine in a UTM Debian VM...

Hope this helps!