search

docker / for-win

Bug reports for Docker Desktop for Windows
https://www.docker.com/products/docker#/windows
1.85k stars 287 forks source link

Docker desktop have no response and the generated log files consume disk space rapidly #13926

Open wohenbushuang opened 6 months ago

wohenbushuang commented 6 months ago

Description

After a long run, sometimes docker desktop has no response, and it generates a lot of log files such as electron-2024-02-21-17.log.xx in C:\Users\admin\AppData\Local\Docker\log\host.

The latest log is like this: 

2024-02-21T05:16:14.953Z info   [HEARTBEAT] containersOpened
2024-02-21T05:16:14.954Z info   [HTTP] 200: POST /usage
2024-02-21T05:16:15.026Z info   [HTTP] 200: GET /ping
2024-02-21T05:16:15.032Z info   [MENU] Rebuilding with licenseHasBeenAccepted: false and loginRestricted: false
2024-02-21T05:16:15.033Z info   [MENU] Rebuilding with licenseHasBeenAccepted: false and loginRestricted: false
2024-02-21T05:16:15.034Z info   [MENU] Rebuilding with licenseHasBeenAccepted: false and loginRestricted: false
2024-02-21T05:16:15.034Z info   [MENU] Rebuilding with licenseHasBeenAccepted: true and loginRestricted: false
2024-02-21T05:16:15.035Z info   [MENU] Rebuilding with licenseHasBeenAccepted: true and loginRestricted: false
2024-02-21T05:16:15.038Z error  Unhandled uncaught exception : {}
2024-02-21T05:16:15.043Z info   [HTTP] 200: GET /versions
2024-02-21T05:16:15.043Z info   [MENU] Rebuilding with licenseHasBeenAccepted: true and loginRestricted: false
2024-02-21T05:16:15.044Z info   [HTTP] 200: GET /backend/state
2024-02-21T05:16:15.045Z info   [MENU] Rebuilding with licenseHasBeenAccepted: true and loginRestricted: false
2024-02-21T05:16:15.046Z info   [HTTP] 200: GET /docker
2024-02-21T05:16:15.048Z info   [MENU] Rebuilding with licenseHasBeenAccepted: true and loginRestricted: false
2024-02-21T05:16:15.048Z info   [HTTP] 200: GET /app/settings
2024-02-21T05:16:15.049Z info   [MENU] Adding items for extensions { newExtensionsAvailable: 0, updatesAvailable: 0 }
2024-02-21T05:16:15.050Z info   [MENU] Rebuilding with licenseHasBeenAccepted: true and loginRestricted: false
2024-02-21T05:16:15.051Z info   [HTTP] 200: GET /features
2024-02-21T05:16:15.051Z info   [HTTP] 200: GET /install/check
2024-02-21T05:16:15.052Z info   [MENU] Adding items for extensions { newExtensionsAvailable: 0, updatesAvailable: 0 }
2024-02-21T05:16:15.053Z info   [MENU] Rebuilding with licenseHasBeenAccepted: true and loginRestricted: false
2024-02-21T05:16:15.054Z info   [HTTP] 200: GET /kubernetes
2024-02-21T05:16:15.055Z info   [MENU] Adding items for extensions { newExtensionsAvailable: 0, updatesAvailable: 0 }
2024-02-21T05:16:15.055Z info   [MENU] Rebuilding with licenseHasBeenAccepted: true and loginRestricted: false
2024-02-21T05:16:15.055Z info   [HTTP] 200: GET /features
2024-02-21T05:16:15.057Z info   [MENU] Adding items for extensions { newExtensionsAvailable: 0, updatesAvailable: 0 }
2024-02-21T05:16:15.057Z info   [MENU] Rebuilding with licenseHasBeenAccepted: true and loginRestricted: false
2024-02-21T05:16:15.061Z info   [HTTP] 200: GET /store
2024-02-21T05:16:15.075Z info   [HTTP] 200: GET /store
2024-02-21T05:16:15.079Z info   [MENU] Adding items for extensions { newExtensionsAvailable: 0, updatesAvailable: 0 }
2024-02-21T05:16:15.079Z info   [MENU] Rebuilding with licenseHasBeenAccepted: true and loginRestricted: false
2024-02-21T05:16:15.101Z error  Unhandled uncaught exception : {}
2024-02-21T05:16:15.102Z error  Unhandled uncaught exception : {}
2024-02-21T05:16:15.102Z error  Unhandled uncaught exception : {}
2024-02-21T05:16:15.103Z error  Unhandled uncaught exception : {}
2024-02-21T05:16:15.103Z error  Unhandled uncaught exception : {}
2024-02-21T05:16:15.103Z error  Unhandled uncaught exception : {}
2024-02-21T05:16:15.103Z error  Unhandled uncaught exception : {}
2024-02-21T05:16:15.104Z error  Unhandled uncaught exception : {}
2024-02-21T05:16:15.104Z error  Unhandled uncaught exception : {}
2024-02-21T05:16:15.104Z error  Unhandled uncaught exception : {}
2024-02-21T05:16:15.104Z error  Unhandled uncaught exception : {}
2024-02-21T05:16:15.105Z error  Unhandled uncaught exception : {}
......

Then it only has the same Unhandled uncaught exception in the later log file.

These log files take around 2.7 GB per hour, and they quickly run out of my disk!

Reproduce

run some containers and wait. recently i usually ran containers (ubuntu based) with a lot of memory usage.

Expected behavior

No response

docker version

Client:
 Cloud integration: v1.0.35+desktop.10
 Version:           25.0.3
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        4debf41
 Built:             Tue Feb  6 21:13:02 2024
 OS/Arch:           windows/amd64
 Context:           default

Server: Docker Desktop 4.27.2 (137060)
 Engine:
  Version:          25.0.3
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       f417435
  Built:            Tue Feb  6 21:14:25 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    25.0.3
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1-desktop.4
    Path:     C:\Program Files\Docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.5-desktop.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-compose.exe
  debug: Get a shell into any image or container. (Docker Inc.)
    Version:  0.0.24
    Path:     C:\Program Files\Docker\cli-plugins\docker-debug.exe
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-dev.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.21
    Path:     C:\Program Files\Docker\cli-plugins\docker-extension.exe
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     C:\Program Files\Docker\cli-plugins\docker-feedback.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.0.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-init.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sbom.exe
  scout: Docker Scout (Docker Inc.)
    Version:  v1.4.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
 Containers: 12
  Running: 1
  Paused: 0
  Stopped: 11
 Images: 21
 Server Version: 25.0.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
 Kernel Version: 5.15.133.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 20
 Total Memory: 15.49GiB
 Name: docker-desktop
 ID: b91cf323-d6dc-44b1-8280-56ee16b8cac2
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Registry Mirrors:
  https://docker.mirrors.ustc.edu.cn/
  https://mirror.ccs.tencentyun.com/
  https://dockerproxy.com/
  https://docker.nju.edu.cn/
  https://docker.m.daocloud.io/
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: daemon is not using the default seccomp profile

Diagnostics ID

B3E4D14A-8D70-493F-8663-FB1918D23269/20240222081303

Additional Info

No response

martinberteig commented 6 months ago

I have experienced the same for some time. Its been very on and off so its been difficult to diagnose.

Literally yesterday i figured out what caused this issue in our environment. Not sure if its the same cause as in your scenario, but our issue came from our automation in starting/stopping of Docker Desktop. Do you automate starting/stopping of Docker Desktop?

In our automation we had a bug that started two instances of Docker Desktop in rapid succession. Which in some scenarios caused the dashboard to freeze and we got these errors in the logs as you mentioned above.

I have managed to recreate the issue with the following code. It only occurs in approximately 1 out of 5 times

Get-Process com.docker*| Stop-Process -Force -ErrorAction SilentlyContinue
Get-Process docker* | Stop-Process -Force -ErrorAction SilentlyContinue
Get-Process Docker* | Stop-Process -Force -ErrorAction SilentlyContinue

Start-Sleep 5
Start-Process -FilePath "$Env:ProgramFiles\Docker\Docker\Docker Desktop.exe"
Start-Process -FilePath "$Env:ProgramFiles\Docker\Docker\Docker Desktop.exe"

Edit: I have reproduced this on a fresh updated Win11 VM, manually installed Docker desktop and with no containers

wohenbushuang commented 6 months ago

I don't use any automation. My work process is: turn on my computer -> manually start Docker Desktop -> open WSL and locate to my project -> open vscode from WSL -> reopen in devcontainer -> kill Docker Desktop if it stuck (maybe OOM)

Neorej commented 6 months ago

I have (had) the same issue. The line error Unhandled uncaught exception : {} was being written to logs hundreds of times per second, adding ~18MB of log files every minute. This has apparently been going on for several weeks, and I ended up with 75GB of log files.

Disabling "Docker Desktop Launcher" in Windows task manager -> Startup seems to have fixed the issue.

DanielHabenicht commented 5 months ago

I got the same issues as @Neorej on one of our build servers. We added 96GB in 3 days. Here is the first file of hundreds: electron-2024-03-01-07.log image Interestingly it started on the first of the month.

LKujala commented 4 months ago

The electronxxx.log files have filled my hdd. docker was acting flaky on startup; tried running update with no success. finally manually downloaded installer and reran that to get docker to current version. its now starting fine. however, I have about 110gb of log files in \appdata\local\docker\log\host.

this is the ONLY (??) post I have found referencing these log files, so I am asking the question here: what files can I safely delete?

eg which of these can I delete electron-2024-04-13-06.log electron-2024-04-13-06.log.1 electron-2024-04-13-06.log.2 ... electron-2024-04-13-06.log.24 etc

fyi content of files is "2024-04-12T11:00:00.000Z error Unhandled uncaught exception : {}"

diagnostics id (I think) 3B8749F4-C761-45F4-AA4D-B383BD460D29/20240423205743

fooware commented 3 months ago

We have the same issues, an overflowing amount of electron logs have filled up the system drive one of our build servers, taking it down on several occasions now.

It is not constant, as it doesn't happen all the time according to our disk usage metrics, but after it starts it seems to continue until there is no disk space left.

The Docker Desktop version on it is 4.29.0 (145265), so not the same as the original poster.

jamest-pin commented 3 months ago

Just had this issue on a Mac as well. I had noticed no problems with Docker in weeks prior, but woke up to find my 500GB disk full of 384GB system data, which I managed to track down to docker logs. docker system prune and docker system prune -af didn't do anything to clear them up. I ended up deleting the folder manually.

image

gabe-gfm commented 3 months ago

Apparently this is not only a docker for-win issue.

My organization is also seeing this same behavior on Docker for Mac too.

I just checked and found 180GB of logs in ~/Library/Containers/com.docker.docker/Data/log/host/electron.*.log

024-06-09T21:15:00.509Z error   Unhandled uncaught exception : { message: 'write EPIPE', errno: -32, code: 'EPIPE', syscall: 'write' }
2024-06-09T21:15:00.510Z error  Unhandled uncaught exception : { message: 'write EPIPE', errno: -32, code: 'EPIPE', syscall: 'write' }
2024-06-09T21:15:00.510Z error  Unhandled uncaught exception : { message: 'write EPIPE', errno: -32, code: 'EPIPE', syscall: 'write' }
2024-06-09T21:15:00.510Z error  Unhandled uncaught exception : { message: 'write EPIPE', errno: -32, code: 'EPIPE', syscall: 'write' }

We use k8s for docker. The daemon and client are regularly crashing. It seems like the client could better handle when some thread starts throwing an exception like this.

gabe-gfm commented 3 months ago

Here are my example logs from when things go haywire:

Investigating the logs, the final "normal" log message before bouts of Unhandled uncaught exception is relating to /stats. This correlates with the docker daemon being unavailable from the command line.

My environment has heavy memory usage. Desktop: 4.30.0 (149282) Engine: 26.1.1 Kubernetes: v1.29.2

charliik commented 2 months ago

Same here, W10 Pro, Docker desktop version 4.31.1

DT-BMA commented 2 months ago

This only seems to have happened recently, I've been testing Kubernetes in WSL2-Docker in a VM (Yep, nested virtualization) and after updating this week, it seems the docker desktop spam is intent in filling my VM without an option to disable logging from the client.

image 160GB of wasted space in three days.

2024-06-17T14:44:49.246Z error Unhandled uncaught exception : { message: 'EPIPE: broken pipe, write', errno: -4047, syscall: 'write', code: 'EPIPE' } 2024-06-17T14:44:49.246Z info [SSE] Disconnecting from /kubernetes/events 2024-06-17T14:44:49.247Z info [SSE] Disconnecting from /registry/state 2024-06-17T14:44:49.248Z error Unhandled uncaught exception : { message: 'EPIPE: broken pipe, write', errno: -4047, syscall: 'write', code: 'EPIPE' } Mostly Epipe/broken pipe everywhere.

I use RDP to connect to my VMs, and I feel occasionally however windows handles sign outs / suspends likes to cause me issues with WSL2 + Docker Desktop + Kubernetes. I'll have to see if I can force the Host logs folder to be read only and not have docker desktop overwrite it

jocxfin commented 2 months ago

Yup. Same here. Filled my C-drive and I noticed this only after disk space was at 0 and Docker crashed and got notifications from my down detector. Turns out there were 264 gigabytes of electron log files on my nvme. WIN11E, Docker Desktop v4.31.1, engine 26.1.4.

edit: found out this issue yesterday, and after deleting the log files, there is now 1.2gb of logs files already.

sklakegg commented 2 months ago

Same issue on MAC - 274 GB of log files 💀

match8969 commented 2 months ago

Same issue on MAC M3. 247GB filled with log files.

docker version shows

Client:
 Cloud integration: v1.0.35+desktop.13
 Version:           26.1.1
 API version:       1.45
 Go version:        go1.21.9
 Git commit:        4cf5afa
 Built:             Tue Apr 30 11:44:56 2024
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.30.0 (149282)
 Engine:
  Version:          26.1.1
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.9
  Git commit:       ac2de55
  Built:            Tue Apr 30 11:48:04 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.31
  GitCommit:        e377cd56a71523140ca6ae87e30244719194a521
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
Fellable commented 2 months ago

Same problem, 150gb / month. OC: Windows 11. Version: 4.29.0

sbaugh-rh commented 2 months ago

Same - 175gb / month MacOS Sonoma 14.5

PatrickNausha commented 2 months ago

I can repro on macOS as well.

macOS Sonoma 14.5 Docker Desktop 4.32.0 (157355) Engine: 27.0.3

❯ sudo du -h -d1 /System/Volumes/Data/Users/patrick.nausha/Library/Containers/com.docker.docker/Data/log/host | sort -h
309G    /System/Volumes/Data/Users/patrick.nausha/Library/Containers/com.docker.docker/Data/log/host

Several thousand electron files are in that directory.

❯ ls -l /System/Volumes/Data/Users/patrick.nausha/Library/Containers/com.docker.docker/Data/log/host | grep 'electron-' | wc -l
   16255

The logs contain mostly (exclusively?) these lines which resemble what others in this issue see:

2024-07-10T17:52:16.871Z error  Unhandled uncaught exception : { message: 'write EPIPE', errno: -32, code: 'EPIPE', syscall: 'write' }
trungutt commented 1 month ago

Hi all, we're trying to address this issue internally. Could you please try out this build to see if the issue still persists?

Windows: https://desktop-stage.docker.com/win/main/amd64/159873/Docker%20Desktop%20Installer.exe

I see some Mac users here, these are Mac builds:

Mac M1: https://desktop-stage.docker.com/mac/main/arm64/159873/Docker.dmg Mac Intel: https://desktop-stage.docker.com/mac/main/amd64/159873/Docker.dmg

I'd like to know if one of you have a least reproduction case, or at least any symptom of malfunctioning of Docker Desktop in case this kind of log start to accumulate.

Sorry for any inconvenience.

cainsyake commented 1 month ago

I have the same problem. My docker desktop info: Current version: 4.30.0 (149282)

electron.log content: 2024-07-19T15:11:53.900Z error Unhandled uncaught exception : { message: 'EPIPE: broken pipe, write', errno: -4047, syscall: 'write', code: 'EPIPE' }

DT-BMA commented 1 month ago

I'll be leaving my usual containers running in my RDP'd Win10 VM (nested virt on, WSL2 enabled, Ubuntu 24.04 default attached distro) with the latest version of Docker-Desktop (Engine v27.03, Kubernetes 1.29.2, Desktop V4.32) over the weekend to see what issues come up, if any.

I've not had the same massive log spam use up all my disk space in the past few weeks and I regularly update my docker desktop, but I do think a better option would be to force the electron backend to have a limit of 8GB of logfiles on rotation by default to avoid this entirely?

jamest-pin commented 1 month ago

to have a limit of 8GB of logfiles on rotation by default to avoid this entirely?

Agreed, having a configurable limit with a sensible default ought to be implemented as a protection against this class of issue, in addition to fixing the particular instance.

splundid commented 1 month ago

Can reproduce this on mac Server: Docker Desktop 4.30.0 (149282) . Had to delete more than 280GB of electron logs

citrusjunoss commented 1 month ago

Are there any plans to fix it or is there any way to avoid it?

trungutt commented 1 month ago

Hi all, we're trying to address this issue internally. Could you please try out this build to see if the issue still persists?

Windows: desktop-stage.docker.com/win/main/amd64/159873/Docker%20Desktop%20Installer.exe

I see some Mac users here, these are Mac builds:

Mac M1: desktop-stage.docker.com/mac/main/arm64/159873/Docker.dmg Mac Intel: desktop-stage.docker.com/mac/main/amd64/159873/Docker.dmg

I'd like to know if one of you have a least reproduction case, or at least any symptom of malfunctioning of Docker Desktop in case this kind of log start to accumulate.

Sorry for any inconvenience.

Hi all,

I'd like to reiterate my request above. You could even try the latest release of Docker Desktop (4.33.0) https://www.docker.com/products/docker-desktop/ to see if the issue still persists.

Sorry for any inconvenience.

265chemic commented 1 month ago

Just found this after troubleshooting a full disk issue, also due to many 20481kb electron logs filling the drive. Docker Desktop process was hanging also, required close in task manager then reopen to use. Docker Desktop 4.28.0 (139021) on win10.

kirill-tolmachev commented 3 weeks ago

Updating from 4.27.0 to 4.33.0 seemingly fixed the issue for me.

scottruzal commented 1 week ago

Experiencing this issue now for the first time after updating to Docker Desktop 4.34 on my Windows 10 machine. Had to change my Microsoft pin and sign out of all applications that use Windows login to even log into the machine.

Also have been regularly experiencing errors in starting up Docker Engine with error messages like the following (not sure if they are related):

running engine: waiting for the VM setup to be ready: starting WSL engine: bootstrapping in the main distro: context canceled_

deploying WSL2 distributions ensuring main distro is deployed: checking if main distro is up to date: checking main distro bootstrap version: getting main distro bootstrap version: open \wsl$\docker-desktop\etc\wsl_bootstrap_version: The network name cannot be found. checking if isocache exists: CreateFile \wsl$\docker-desktop-data\isocache\: The network name cannot be found.

running wsl distro proxy in Ubuntu-18.04 distro: exit status 1

stoddn commented 6 days ago

Same on Mac M1 (Sonoma 14.6.1) version 4.34.0.

Electron logs consumed almost 220G in a day and a half, and all kinds of apps started failing because of maxed out disk.

I was able to recover by deleting "~/Library/Containers/com.docker.docker/Data/log/host/electron-*.log.*"

cross posted to https://github.com/docker/for-mac/issues/7348

Howler4695 commented 6 days ago

I'm having the same issue Mac M1 (Sonoma 14.6.1) docker v27.2.0

klichukb commented 4 days ago

Same issue, M1 Sonoma 14.6.1

klichukb commented 4 days ago

Updated docker desktop to 4.34, I think that resolved the issue (docker engine 27.2.0)