[BUG] Docker compose starting lags

ihor-sviziev commented 6 months ago

Description

I noticed that my environment started very slowly - like five containers were re-created (w/o building or pulling) for about 2 minutes, while in the past, it was about 5-10 seconds. It literally shows that it creating container one by one with delay of about 10+ seconds each. Investigation came that actually not depending in which folder - docker compose config, docker compose up -d and even docker compose commands time to time becoming super-slow.

Environment: Macbook Pro with M1 Pro CPU MacOS 14.2.1 (23C71) Docker Desktop for Mac 4.26.1 (131620)

Steps To Reproduce

Run Docker Desktop for Mac
Run the time docker compose command

Expected result command should execute under 1 second

Actual result command executes for 10 seconds (we're getting output immediately, but then waiting for 10 seconds. The end of the command output is following:

...
Run 'docker compose COMMAND --help' for more information on a command.
docker compose  0.07s user 0.03s system 1% cpu 10.102 total

Compose Version

Docker Compose version v2.23.3-desktop.2

Docker Environment

Client:
 Version:    24.0.7
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.0-desktop.2
    Path:     /Users/ihor/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.23.3-desktop.2
    Path:     /Users/ihor/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/ihor/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.21
    Path:     /Users/ihor/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  0.1
    Path:     /Users/ihor/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.10
    Path:     /Users/ihor/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/ihor/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/ihor/.docker/cli-plugins/docker-scan
  scout: Docker Scout (Docker Inc.)
    Version:  v1.2.0
    Path:     /Users/ihor/.docker/cli-plugins/docker-scout

Server:
 Containers: 11
  Running: 11
  Paused: 0
  Stopped: 0
 Images: 11
 Server Version: 24.0.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
 runc version: v1.1.10-0-g18a0cb0
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.5.11-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 10
 Total Memory: 7.663GiB
 Name: docker-desktop
 ID: 8a4be415-badd-461e-84e9-0ce5f4444362
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

Anything else?

Notes:

Full uninstall of Docker Desktop, re-installing it + reboot the computer helped for some period, and then it appeared again.
Commands like docker ps, docker info and docker compose version don't have any delay
When the Docker Desktop is stopped - the command returns the output and finishes immediately\
Computer isn't under load when the issue happening, the CPU load is up to 20%, and memory up to 60%
The issue starts reproducing randomly for me and my colleague
Downgrading to previous versions of Docker Desktop didn't help
Important: the issue become self-fixed after about 2-3 hours, and time docker compose moves back to normal, here is the screenshot when it become self-fixed

Diagnostic ID E27C59CC-514B-4202-94C5-89872498DCEB/20231227103223

SerhiiMandrykin commented 6 months ago

Hi. I'm facing the same issue.

isareds commented 6 months ago

Hi. Me too, I'm facing the same issue.

aleksinoleg commented 6 months ago

Hi! I have the same issue. Please fix it ASAP.

mariosanchez23 commented 6 months ago

The same! it started just after upgrading Docker Desktop!

ihor-sviziev commented 6 months ago

@milas, were you able to reproduce the issue? Maybe you know some workarounds against this issue? Or maybe you need some addition details from me?

Feng-Shui commented 6 months ago

Seeing similar problems, composer commands seem to hang for 10 to 20 seconds before doing anything, then execute the command very slowly. A compose up command that would normally execute in a second or two is now taking upwards of 30 seconds.

asaken1021 commented 6 months ago

I have the same issue. compose up command taking about 40 seconds to complete.

ihor-sviziev commented 6 months ago

Hi @milas, I just noticed that the issue started reproducing for me just after reboot even when Docker Desktop isn't started yet:

Interesting that just now after starting Docker Desktop, the issue stopped reproducing

Maybe this additional info will help you to find issue faster

mariosanchez23 commented 6 months ago

The same! it started just after upgrading Docker Desktop!

In my case it was solved with a reboot

ihor-sviziev commented 5 months ago

The interesting thing, after upgrading Docker Desktop to v4.27.0 (135262), I still have this issue, but the time docker compose command now takes about 2 seconds, which is 5 times better, but still not what we expect.

time docker compose
...
docker compose  0.06s user 0.03s system 4% cpu 2.086 total

ihor-sviziev commented 5 months ago

@glours maybe you can also look at this issue? This is very annoying :( Unfortunately, with latest release, it still takes about 2 seconds for running just docker compose w/o any params. Here is a short video that shows how it works (tested on latest release binary): https://drive.google.com/file/d/1VBtj9qPBfJ0NyzggTv9gxRajO2bY-tRW/view?usp=drive_link In about 1 hour the issue self-fixed, here is a new video: https://drive.google.com/file/d/1aob0aHG3wwOow1453QzZJsc07frkU73S/view?usp=sharing

glours commented 5 months ago

It looks like you're reaching the telemetry timeout, I suspect something on the DD side, @milas did improvements on Docker Desktop side but they aren't merged ATM. Next time this behaviour happens can you check if you have otlp.grpc.sock and otlp-public.grpc.sock files in your ~/.docker/run directory please?

ihor-sviziev commented 5 months ago

@glours, just checked - yes, these 2 files are there when the issue reproduces:

➜  ~ ll ~/.docker/run                                                         ✔
total 0
srwxr-xr-x  1 ihor  staff     0B Jan 30 16:06 docker-cli-api.sock
srwxr-xr-x  1 ihor  staff     0B Jan 30 16:06 docker.sock
srwxr-xr-x  1 ihor  staff     0B Jan 30 16:06 otlp-public.grpc.sock
srwxr-xr-x  1 ihor  staff     0B Jan 30 16:06 otlp.grpc.sock

Also, I checked, I have Send usage statistics enabled.

So far, disabling usage statistics didn't change anything, also removing these 2 sock files didn't change. Note: i used the following commands:

rm ~/.docker/run/otlp-public.grpc.sock
rm ~/.docker/run/otlp.grpc.sock

dave-shawley commented 4 months ago

Is this related to (or the same issue as) https://github.com/docker/compose/issues/11437 ?

milas commented 4 months ago

@dave-shawley It's possible they're related. Are you on Docker Desktop 4.28? (Click "Check for Updates" and it should show up if it hasn't automatically.)

If you still see the issue after upgrading, can you upload a diag and share the ID here?

/Applications/Docker.app/Contents/MacOS/com.docker.diagnose gather -upload

dave-shawley commented 4 months ago

I’m still on 4.27.2…. our packages are managed on my work laptop and 4.28 isn’t available yet. It should show up in a day or two. I’ll check back in after I upgrade. 🤞

nocive commented 4 months ago

~~Likely the same issue I started observing with versions >= 2.24.4. A simple test is to go back to 2.24.3 and observe that the same startup delay for every single docker compose command no longer occurs.~~

I have so far failed to isolate the problem, but with our project setup it only occurs when COMPOSE_FILE is set in the environment (our project uses a 2 docker-compose.yml file setup and we use COMPOSE_FILE for that).

EDIT: scratch that, this is likely a different bug since I'm not using docker desktop nor do I see such a delay with versions <=2.24.3.

dave-shawley commented 4 months ago

@milas I did a full clean and reinstall of Docker Desktop and the slowness that I was seeing has disappeared 🤨

I think that I hadn't removed some of the cruft in ~/Library/Containers/com.docker.docker the last time that I uninstalled. I should note that /Applications/Docker.app/Contents/MacOS/uninstall failed in a weird way. I cleaned up the stuff hanging out in ~/Library, rebooted, and reinstalled. Now it works ¯_(ツ)_/¯

docker / compose