Closed boda-sh closed 1 year ago
@bpogodzinski have you tried to increase your Memory settings in Docker? I increased them from 2GB up to 8GB and that fixed the problem for me.
Generally speaking, this issue seems to happen when the containers require more memory than the configured available memory in Docker and then stuff just hangs.
We had this issue and it appears (for us) to be related to a named volume with a lot of files. I don't understand it, but it is the case for us that a docker-compose (edited for brevity) that has a service:
serviceA:
...
volumes:
- serviceA_volume: /srvA/folder
volumes:
- serviceA_volume:
Inside the Dockerfile for serviceA is the seemingly harmless and ineffectual command:
...
RUN mkdir -p /srvA/folder && chown -R user /srvA/folder
...
Notice that this changes the owner recursively in /srvA/folder which in the named volume is a large filesystem with 100K's of files. However, this happens when the image is built and that folder is empty. It appears using the named volume inherits the permissions of the image local file and then proceeds to change the named volumes permissions.
This is pretty edge and probably not the same problem everyone else is having but it was our problem (toggling the line toggles the error). The upshot is that this http timeout is probably resulting from multiple causes.
Restarting docker never solved the issue in my case, increasing the resources definitely did.
From my experience this problem often arises on small cloud instances where the amount of RAM is perfectly fine during regular functioning but proves insufficient during docker or docker-compose operations. You could easily increase the RAM, but it would probably drastically increase the cost of a small VM.
In each case, adding a swap partition or even a swap file solved this issue for me!
Just occured to me on a raspberry pi. No volume with huge amount of files or anything. Actually I've been spawning these containers on that raspberry for a while now (a year or two lol). Not sure what changed. Seems a bit "out of the blue".
Problem still appears on docker desktop 2.2.0.3 on MacOs 🙁
I resolved my issue with the following commands:
docker volume prune
docker system prune
(only one of these commands might be enough, but cannot reproduce for the moment...)
I resolved my issue with the following commands:
docker volume prune
docker system prune
(only one of these commands might be enough, but cannot reproduce for the moment...)
@amaumont 's solution helped, although I think this would continue coming back overtime. As everyone else has said, restarting docker is not a proper solution, it's a bandaid.
We are having multiple issues with docker-compose, too.
After setting MaxSessions 500
in sshd_config (see #6463) we now also get read time outs.
Setting both timeouts to 120 seconds resolved the issue for the next DOCKER_HOST=xxx@yyy docker-compose up -d
run.
During the second run the machine load went as high as 30 (sic!) before the docker-compose command failed due to timeouts. A docker restart did does not solve this problem, not even temporarily.
The Server is an AWS EC2 instance with enough CPU/Disk/NetIO etc, the compose file includes 1 traefik and 3 services with mailhog, so nothing special here. Running docker-compose up -d
with the same docker-compose.yml file directly on the server works reliably and as expected.
Running with --verbose shows over thousand consecutive lines containing compose.parallel.feed_queue: Pending: set()
.
I will try to rsync the docker-compose file to the remote server and run docker-compose directly on that machine as a workaround.
For me, it helped to just restart docker.
Happens pretty often for me when trying to push to my private registry from bitbucket pipelines. Works well when pushing from local PC tho. Restarting docker could help for a while, however this "while" lasts for 10 min max :c
upd. setting DOCKER_CLIENT_TIMEOUT
and COMPOSE_HTTP_TIMEOUT
seemed to help, but I don't know for how long
I started getting these since switching to Docker Edge with Caching on
This has been happening pretty consistently for me since I started using Docker 2-3 years ago. After a container has been running for a while, it becomes a zombie and the entire Docker engine needs to be restarted for things to become responsive again. This feels like a resource leak of some kind, since idle time seems to be very relevant for the experienced behaviour.
If no containers are running, or they only run for a short amount of time, everything seems to be working fine for days or weeks. But as soon as I let a container run for a few hours, it becomes unresponsive, I have to force-stop it in the command line and any attempt at communicating with docker
or docker-compose
just fails with a timeout. A restart is the only working solution.
Output of docker-compose version
docker-compose version 1.25.5, build 8a1c60f6
docker-py version: 4.1.0
CPython version: 3.7.5
OpenSSL version: OpenSSL 1.1.1f 31 Mar 2020
Output of docker version
Client: Docker Engine - Community
Version: 19.03.8
API version: 1.40
Go version: go1.12.17
Git commit: afacb8b
Built: Wed Mar 11 01:21:11 2020
OS/Arch: darwin/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.8
API version: 1.40 (minimum version 1.12)
Go version: go1.12.17
Git commit: afacb8b
Built: Wed Mar 11 01:29:16 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.2.13
GitCommit: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683
Output of docker-compose config
services:
portal:
container_name: developer_portal
image: swedbankpay/jekyll-plantuml:1.3.8
ports:
- published: 4000
target: 4000
- published: 35729
target: 35729
volumes:
- .:/srv/jekyll:rw
- ./.bundle:/usr/local/bundle:rw
version: '3.7'
macOS Mojave 10.14.6.
I faced same issue, even I increased resource from 4GB RAM, 1GB swap to 6GB RAM, 2GB swap.
I am also facing the same issue
also having same issue
I've been facing the same issue on Ubuntu 18.04 LTS (8 GB RAM) using HTTPS.
I'm able to spawn containers with docker-compose up
, however once deployed I'm unable to stop containers with docker-compose down
. Restarting the docker daemon or rebooting the VM have proven to be ineffective. Adding timeout environment variables (DOCKER_CLIENT_TIMEOUT
, COMPOSE_HTTP_TIMEOUT
) also didn't do anything.
I'm able to interact with and stop containers individually, I can inspect containers, attach to them, and anything else, but I cannot stop or kill them using docker-compose
command.
The error message is always the same:
msg: 'Error stopping project - HTTPSConnectionPool(host=[ommited], port=2376): Read timed out. (read timeout=120)
I was having the same issue when I had the following in my docker-compose.yml:
logging:
driver: "json-file"
options:
max-size: 100m
max-file: 10
The error was gone when I added quotes around "10". This is stated in docs that the values for max-file and max-size must be string, but still. The error message is quite misleading.
I was having the same issue when I had the following in my docker-compose.yml:
logging: driver: "json-file" options: max-size: 100m max-file: 10
The error was gone when I added quotes around "10". This is stated in docs that the values for max-file and max-size must be string, but still. The error message is quite misleading.
You save my day. Thank you so much!
I was having the same issue when I had the following in my docker-compose.yml:
logging: driver: "json-file" options: max-size: 100m max-file: 10
The error was gone when I added quotes around "10". This is stated in docs that the values for max-file and max-size must be string, but still. The error message is quite misleading.
I'm configuring the logging driver at docker daemon level. I'm using fluentd as my logging-driver, so unfortunately this fix won't work for me. =/
tried this
export DOCKER_CLIENT_TIMEOUT=120 export COMPOSE_HTTP_TIMEOUT=120
and it seems to fix the issue for now
Other solutions people mentioned in this thread:
- Restart Docker
- Increase Docker CPU & memory
Well, nothing worked for me, except the timeout option, kudos to you.
I'm getting this since I started to use an NFS mounted directory inside one of my containers. That NFS mounted directory is on a slow link (in a remote location that has a low bandwidth connection). Could that be the problem?
I'm experiencing this very frequently on Mac, Docker 2.4.0.0, in two different projects with different docker-compose.yml configs. I don't recall it ever happening before ~1 week ago which is when I upgraded to 2.4.0.0. Is there a regression?
I've tried increasing the timeout to 600, increasing RAM to 16GB & swap to 4GB, restarting Docker, restarting my entire Macbook, nothing seems to work, except randomly trying again and again then it will occasionally work. But then the next time I need to restart or rebuild a container, same problem :(
Started seeing this with 2.4.0.0 on Mac as well. Workaround for me is to restart docker but will run into it again later.
Same here! With update to 2.4.0 our setups sometimes do not start at all with the mentioned Read timed out.
errors, sometimes only some containers start up, others throw this error. I am already thinking about a downgrade!
Just to mention: This issue affects both setups using NFS shares as well as projects using "normal" mounted volumes
Same issue here, also on mac and after the 2.4.0 update. I'm currently trying if downgrading helps.
Update: downgrading to the previous version, deleting cache and rebuilding fixes the issue.
I also recently started seeing this issue (Mac, 2.4.0.0), when I never saw it before. Running docker image prune
made the problem go away for a couple of days, but now it's back again.
Also started having frequently this timeout error since the 2.4.0 update (on Mac OS Mojave 10.14.5)
Also seeing this with increased frequency since updating to Docker Desktop 2.4.0.0 (48506) on MacOS Catalina.
I get the same timeouts issues since 2.4.0.0 on Mac OS. I never had this issue before. I tried the edge build 2.4.1.0 (48583) but I still have the same issue.
I got same issue and rebooting docker fixed it for MacOs Catalina(10.15.5) and docker version 2.4.0.0
Same here, didn't have the problem before updating to Docker desktop 2.4.0.0. Restarting Docker desktop works, but it's just a workaround.
Same here, also starting with v2.4.0
Update: downgrading to the previous version, deleting cache and rebuilding fixes the issue.
Will try that. Not even sure how it's done. I assume it's by uninstalling and downloading an earlier version?
Yes I uninstalled the 2.4 and downloaded/reinstalled the 2.3. Now it works, I can start my containers as usual. I got the 2.3 from there: https://docs.docker.com/docker-for-mac/release-notes/#docker-desktop-community-2302
Yup, can confirm it made the difference for me too. Definitely v2.4 is to blame here somehow.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).
How 1Gbps is a slow network, exactly?
Downgrading worked for me as well. For those managing Docker via Homebrew
brew uninstall docker
brew install https://raw.githubusercontent.com/Homebrew/homebrew-cask/9da3c988402d218796d1f962910e1ef3c4fca1d3/Casks/docker.rb
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).
How 1Gbps is a slow network, exactly?
In my case this happened due to an NFS mounted network drive. The "slow" network speed's root cause was the use of NFS not the physical link speed. But it definitely shows there is a problem in the implementation and I would be surprised if changing HTTP_TIMEOUT will solve it.
Same here. Significant slowdown in container creation, resulting in the aforementioned HTTP timeout error on Docker for Mac v2.4. Setting COMPOSE_HTTP_TIMEOUT=120 worked, but the container creation slowness is still a new issue. Downgrading to v2.3 also fixes this.
I can confirm the same problem since I installed on Docker for Mac v2.4. I can also confirm a significant increase of RAM and CPU consumption even in idle moments, just with Docker daemon running. But I guess it has nothing to with compose package itself.
I had this same problem. Uninstalled 2.40 and installed 2.3 from the link mentioned by @ddesrousseaux and I no longer have super slowness or timeouts with starting containers.
https://docs.docker.com/docker-for-mac/release-notes/#docker-desktop-community-2302
This problem still exists in Docker v. 2.4.3.0
.
I've also downgraded to 2.3 from 2.4 to workaround the massive slowness issues in the 2.4 release. Happy to provide whatever logs might be useful to debug what's going on here.
Echoing the above, this started happening in 2.4.2.x for me. Something changed in the upgrade from 2.3.
I made some test in a Linux enviroment, and had a similar problem. I installed the latest docker-compose binary (v1.27.4) and had the same timeout problem you guys are reporting.
After downgrading to 1.27.2, the same available in Docker for Mac 2.3, the problem has disappeared.
Same issue with the current version on Ubuntu 20.04.
My problem was that I installed docker and docker-compose with snap and apt. I uninstalled them, rebooted and then followed the official install instructions at https://docs.docker.com/engine/install/ubuntu/ and https://docs.docker.com/compose/install/
I'm still experiencing frequently timeout errors since the 2.4.0 update that are still not fixed in 2.5.0
Yep, same here. It was working fine for me for the past 2 years. But 2 months ago suddenly when ever i want to 1 instance andd start another docker project it throws : for apache UnixHTTPConnectionPool(host='localhost', port=None): Read timed out.
Restarting Docker fixes the issue. But is a real pain when i have to switch between projects multiple times in 1 day
Hitting same issue since 2.4, 300% cpu at all times, 2.5 didn't help, downgraded back to 2.3 and things are okay. This on latest macbook w/ i7 cpu and 32g ram
I've just upgraded to last Docker for Mac version (v2.5.0.1) and the problem seems to be solved.
No more UnixHTTPConnection
error, and no more 100% CPU use.
Not sure if anyone else can confirm that.
Hi since yesterday I've been running into this error while doing
docker-compose up
Full Error Message
Docker Version Docker for Mac: 1.12.0-a (Build 11213) Machine info MacBook Air (13-inch, Early 2015) Processor: 1.6 GHz i5 Memory: 4GB 1600 MHz DDR3 macOS: Version 10.11.6 (Build 15G1004)
Attempts