docker / compose

Define and run multi-container applications with Docker
https://docs.docker.com/compose/
Apache License 2.0
33.87k stars 5.21k forks source link

UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60) #3927

Closed boda-sh closed 1 year ago

boda-sh commented 8 years ago

Hi since yesterday I've been running into this error while doing docker-compose up

Full Error Message

Device-Tracker $ docker-compose up
Creating device-tracker-db
Creating device-tracker

ERROR: for web  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
Traceback (most recent call last):
  File "<string>", line 3, in <module>
  File "compose/cli/main.py", line 61, in main
  File "compose/cli/main.py", line 113, in perform_command
  File "contextlib.py", line 35, in __exit__
  File "compose/cli/errors.py", line 56, in handle_connection_errors
TypeError: log_timeout_error() takes exactly 1 argument (0 given)
docker-compose returned -1

Docker Version Docker for Mac: 1.12.0-a (Build 11213) Machine info MacBook Air (13-inch, Early 2015) Processor: 1.6 GHz i5 Memory: 4GB 1600 MHz DDR3 macOS: Version 10.11.6 (Build 15G1004)

Attempts

bitbrain commented 4 years ago

@bpogodzinski have you tried to increase your Memory settings in Docker? I increased them from 2GB up to 8GB and that fixed the problem for me.

Generally speaking, this issue seems to happen when the containers require more memory than the configured available memory in Docker and then stuff just hangs.

ajennings6767 commented 4 years ago

We had this issue and it appears (for us) to be related to a named volume with a lot of files. I don't understand it, but it is the case for us that a docker-compose (edited for brevity) that has a service:

   serviceA:
        ...
        volumes:
            - serviceA_volume: /srvA/folder

   volumes:
       - serviceA_volume:

Inside the Dockerfile for serviceA is the seemingly harmless and ineffectual command:

...
RUN mkdir -p /srvA/folder && chown -R user /srvA/folder
...

Notice that this changes the owner recursively in /srvA/folder which in the named volume is a large filesystem with 100K's of files. However, this happens when the image is built and that folder is empty. It appears using the named volume inherits the permissions of the image local file and then proceeds to change the named volumes permissions.

This is pretty edge and probably not the same problem everyone else is having but it was our problem (toggling the line toggles the error). The upshot is that this http timeout is probably resulting from multiple causes.

ntopulos commented 4 years ago

Restarting docker never solved the issue in my case, increasing the resources definitely did.

From my experience this problem often arises on small cloud instances where the amount of RAM is perfectly fine during regular functioning but proves insufficient during docker or docker-compose operations. You could easily increase the RAM, but it would probably drastically increase the cost of a small VM.

In each case, adding a swap partition or even a swap file solved this issue for me!

JnMik commented 4 years ago

Just occured to me on a raspberry pi. No volume with huge amount of files or anything. Actually I've been spawning these containers on that raspberry for a while now (a year or two lol). Not sure what changed. Seems a bit "out of the blue".

michalmisiewicz commented 4 years ago

Problem still appears on docker desktop 2.2.0.3 on MacOs 🙁

amaumont commented 4 years ago

I resolved my issue with the following commands: docker volume prune docker system prune (only one of these commands might be enough, but cannot reproduce for the moment...)

jonnysandu commented 4 years ago

I resolved my issue with the following commands: docker volume prune docker system prune (only one of these commands might be enough, but cannot reproduce for the moment...)

@amaumont 's solution helped, although I think this would continue coming back overtime. As everyone else has said, restarting docker is not a proper solution, it's a bandaid.

vertexvaar commented 4 years ago

We are having multiple issues with docker-compose, too.

After setting MaxSessions 500 in sshd_config (see #6463) we now also get read time outs. Setting both timeouts to 120 seconds resolved the issue for the next DOCKER_HOST=xxx@yyy docker-compose up -d run.

During the second run the machine load went as high as 30 (sic!) before the docker-compose command failed due to timeouts. A docker restart did does not solve this problem, not even temporarily. The Server is an AWS EC2 instance with enough CPU/Disk/NetIO etc, the compose file includes 1 traefik and 3 services with mailhog, so nothing special here. Running docker-compose up -d with the same docker-compose.yml file directly on the server works reliably and as expected. Running with --verbose shows over thousand consecutive lines containing compose.parallel.feed_queue: Pending: set().

I will try to rsync the docker-compose file to the remote server and run docker-compose directly on that machine as a workaround.

nikita-fuchs commented 4 years ago

For me, it helped to just restart docker.

Alveona commented 4 years ago

Happens pretty often for me when trying to push to my private registry from bitbucket pipelines. Works well when pushing from local PC tho. Restarting docker could help for a while, however this "while" lasts for 10 min max :c

upd. setting DOCKER_CLIENT_TIMEOUT and COMPOSE_HTTP_TIMEOUT seemed to help, but I don't know for how long

jasonwilliams commented 4 years ago

I started getting these since switching to Docker Edge with Caching on

asbjornu commented 4 years ago

This has been happening pretty consistently for me since I started using Docker 2-3 years ago. After a container has been running for a while, it becomes a zombie and the entire Docker engine needs to be restarted for things to become responsive again. This feels like a resource leak of some kind, since idle time seems to be very relevant for the experienced behaviour.

If no containers are running, or they only run for a short amount of time, everything seems to be working fine for days or weeks. But as soon as I let a container run for a few hours, it becomes unresponsive, I have to force-stop it in the command line and any attempt at communicating with docker or docker-compose just fails with a timeout. A restart is the only working solution.

Output of docker-compose version

docker-compose version 1.25.5, build 8a1c60f6
docker-py version: 4.1.0
CPython version: 3.7.5
OpenSSL version: OpenSSL 1.1.1f  31 Mar 2020

Output of docker version

Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b
 Built:             Wed Mar 11 01:21:11 2020
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       afacb8b
  Built:            Wed Mar 11 01:29:16 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker-compose config

services:
  portal:
    container_name: developer_portal
    image: swedbankpay/jekyll-plantuml:1.3.8
    ports:
    - published: 4000
      target: 4000
    - published: 35729
      target: 35729
    volumes:
    - .:/srv/jekyll:rw
    - ./.bundle:/usr/local/bundle:rw
version: '3.7'

macOS Mojave 10.14.6.

duyngha commented 4 years ago

I faced same issue, even I increased resource from 4GB RAM, 1GB swap to 6GB RAM, 2GB swap.

pupattan commented 4 years ago

I am also facing the same issue

armmmo commented 4 years ago

also having same issue

pisces-period commented 4 years ago

I've been facing the same issue on Ubuntu 18.04 LTS (8 GB RAM) using HTTPS.

I'm able to spawn containers with docker-compose up, however once deployed I'm unable to stop containers with docker-compose down. Restarting the docker daemon or rebooting the VM have proven to be ineffective. Adding timeout environment variables (DOCKER_CLIENT_TIMEOUT, COMPOSE_HTTP_TIMEOUT) also didn't do anything.

I'm able to interact with and stop containers individually, I can inspect containers, attach to them, and anything else, but I cannot stop or kill them using docker-compose command.

The error message is always the same:

msg: 'Error stopping project - HTTPSConnectionPool(host=[ommited], port=2376): Read timed out. (read timeout=120)
eng1neer commented 4 years ago

I was having the same issue when I had the following in my docker-compose.yml:

logging:
      driver: "json-file"
      options:
        max-size: 100m
        max-file: 10

The error was gone when I added quotes around "10". This is stated in docs that the values for max-file and max-size must be string, but still. The error message is quite misleading.

polly3d commented 4 years ago

I was having the same issue when I had the following in my docker-compose.yml:

logging:
      driver: "json-file"
      options:
        max-size: 100m
        max-file: 10

The error was gone when I added quotes around "10". This is stated in docs that the values for max-file and max-size must be string, but still. The error message is quite misleading.

You save my day. Thank you so much!

pisces-period commented 4 years ago

I was having the same issue when I had the following in my docker-compose.yml:

logging:
      driver: "json-file"
      options:
        max-size: 100m
        max-file: 10

The error was gone when I added quotes around "10". This is stated in docs that the values for max-file and max-size must be string, but still. The error message is quite misleading.

I'm configuring the logging driver at docker daemon level. I'm using fluentd as my logging-driver, so unfortunately this fix won't work for me. =/

princeblr commented 4 years ago

tried this

export DOCKER_CLIENT_TIMEOUT=120
export COMPOSE_HTTP_TIMEOUT=120

and it seems to fix the issue for now

Other solutions people mentioned in this thread:

  • Restart Docker
  • Increase Docker CPU & memory

Well, nothing worked for me, except the timeout option, kudos to you.

dg1sek commented 4 years ago

I'm getting this since I started to use an NFS mounted directory inside one of my containers. That NFS mounted directory is on a slow link (in a remote location that has a low bandwidth connection). Could that be the problem?

mariaines commented 4 years ago

I'm experiencing this very frequently on Mac, Docker 2.4.0.0, in two different projects with different docker-compose.yml configs. I don't recall it ever happening before ~1 week ago which is when I upgraded to 2.4.0.0. Is there a regression?

I've tried increasing the timeout to 600, increasing RAM to 16GB & swap to 4GB, restarting Docker, restarting my entire Macbook, nothing seems to work, except randomly trying again and again then it will occasionally work. But then the next time I need to restart or rebuild a container, same problem :(

KieranHarper commented 4 years ago

Started seeing this with 2.4.0.0 on Mac as well. Workaround for me is to restart docker but will run into it again later.

hctom commented 4 years ago

Same here! With update to 2.4.0 our setups sometimes do not start at all with the mentioned Read timed out. errors, sometimes only some containers start up, others throw this error. I am already thinking about a downgrade!

Just to mention: This issue affects both setups using NFS shares as well as projects using "normal" mounted volumes

FrankHouwelingCastor commented 4 years ago

Same issue here, also on mac and after the 2.4.0 update. I'm currently trying if downgrading helps.

Update: downgrading to the previous version, deleting cache and rebuilding fixes the issue.

Storyyeller commented 4 years ago

I also recently started seeing this issue (Mac, 2.4.0.0), when I never saw it before. Running docker image prune made the problem go away for a couple of days, but now it's back again.

melalj commented 4 years ago

Also started having frequently this timeout error since the 2.4.0 update (on Mac OS Mojave 10.14.5)

khalwat commented 4 years ago

Also seeing this with increased frequency since updating to Docker Desktop 2.4.0.0 (48506) on MacOS Catalina.

ddesrousseaux commented 4 years ago

I get the same timeouts issues since 2.4.0.0 on Mac OS. I never had this issue before. I tried the edge build 2.4.1.0 (48583) but I still have the same issue.

gulistanboylu commented 4 years ago

I got same issue and rebooting docker fixed it for MacOs Catalina(10.15.5) and docker version 2.4.0.0

remigarcia commented 4 years ago

Same here, didn't have the problem before updating to Docker desktop 2.4.0.0. Restarting Docker desktop works, but it's just a workaround.

gnapse commented 4 years ago

Same here, also starting with v2.4.0

Update: downgrading to the previous version, deleting cache and rebuilding fixes the issue.

Will try that. Not even sure how it's done. I assume it's by uninstalling and downloading an earlier version?

ddesrousseaux commented 4 years ago

Yes I uninstalled the 2.4 and downloaded/reinstalled the 2.3. Now it works, I can start my containers as usual. I got the 2.3 from there: https://docs.docker.com/docker-for-mac/release-notes/#docker-desktop-community-2302

gnapse commented 4 years ago

Yup, can confirm it made the difference for me too. Definitely v2.4 is to blame here somehow.

wbednarski commented 4 years ago

If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).

How 1Gbps is a slow network, exactly?

jimmynotjim commented 4 years ago

Downgrading worked for me as well. For those managing Docker via Homebrew

brew uninstall docker
brew install https://raw.githubusercontent.com/Homebrew/homebrew-cask/9da3c988402d218796d1f962910e1ef3c4fca1d3/Casks/docker.rb
dg1sek commented 4 years ago

If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).

How 1Gbps is a slow network, exactly?

In my case this happened due to an NFS mounted network drive. The "slow" network speed's root cause was the use of NFS not the physical link speed. But it definitely shows there is a problem in the implementation and I would be surprised if changing HTTP_TIMEOUT will solve it.

stoutput commented 4 years ago

Same here. Significant slowdown in container creation, resulting in the aforementioned HTTP timeout error on Docker for Mac v2.4. Setting COMPOSE_HTTP_TIMEOUT=120 worked, but the container creation slowness is still a new issue. Downgrading to v2.3 also fixes this.

samoritano commented 4 years ago

I can confirm the same problem since I installed on Docker for Mac v2.4. I can also confirm a significant increase of RAM and CPU consumption even in idle moments, just with Docker daemon running. But I guess it has nothing to with compose package itself.

gallak87 commented 3 years ago

I had this same problem. Uninstalled 2.40 and installed 2.3 from the link mentioned by @ddesrousseaux and I no longer have super slowness or timeouts with starting containers.

https://docs.docker.com/docker-for-mac/release-notes/#docker-desktop-community-2302

taras-GA commented 3 years ago

This problem still exists in Docker v. 2.4.3.0.

timdawborn commented 3 years ago

I've also downgraded to 2.3 from 2.4 to workaround the massive slowness issues in the 2.4 release. Happy to provide whatever logs might be useful to debug what's going on here.

kevinquillen commented 3 years ago

Echoing the above, this started happening in 2.4.2.x for me. Something changed in the upgrade from 2.3.

samoritano commented 3 years ago

I made some test in a Linux enviroment, and had a similar problem. I installed the latest docker-compose binary (v1.27.4) and had the same timeout problem you guys are reporting.

After downgrading to 1.27.2, the same available in Docker for Mac 2.3, the problem has disappeared.

GamerBene19 commented 3 years ago

Same issue with the current version on Ubuntu 20.04.

kontrollanten commented 3 years ago

My problem was that I installed docker and docker-compose with snap and apt. I uninstalled them, rebooted and then followed the official install instructions at https://docs.docker.com/engine/install/ubuntu/ and https://docs.docker.com/compose/install/

andrefcgoncalves commented 3 years ago

I'm still experiencing frequently timeout errors since the 2.4.0 update that are still not fixed in 2.5.0

makproductions commented 3 years ago

Yep, same here. It was working fine for me for the past 2 years. But 2 months ago suddenly when ever i want to 1 instance andd start another docker project it throws : for apache UnixHTTPConnectionPool(host='localhost', port=None): Read timed out.

Restarting Docker fixes the issue. But is a real pain when i have to switch between projects multiple times in 1 day

wenbochang commented 3 years ago

Hitting same issue since 2.4, 300% cpu at all times, 2.5 didn't help, downgraded back to 2.3 and things are okay. This on latest macbook w/ i7 cpu and 32g ram

samoritano commented 3 years ago

I've just upgraded to last Docker for Mac version (v2.5.0.1) and the problem seems to be solved. No more UnixHTTPConnection error, and no more 100% CPU use.

Not sure if anyone else can confirm that.