search

docker / for-win

Bug reports for Docker Desktop for Windows
https://www.docker.com/products/docker#/windows
1.86k stars 291 forks source link

Cleanup of Windows containers does not work properly (An attempt was made to create more links on a file than the file system supports) #13859

Open debug-richard opened 10 months ago

debug-richard commented 10 months ago

Description

I am working with custom Gitlab CI runners which use Docker Windows containers for testing.
All hosts use Windows 11 with Docker Desktop installed.
These hosts launch several hundred containers per day with multiple base images, each comprising ~25 GB with ~40 layers. Base image is always mcr.microsoft.com/windows:ltsc2019 .

As with our Linux runners, the hosts clean up all images every night to keep the system clean with this command: system prune --all --force --volumes

Unfortunately, pulling on Windows hosts after cleanup fails every ~2 weeks with the following error message: failed to register layer: re-exec error: exit status 1: output: link \\?\C:\ProgramData\Docker\windowsfilter\f62f5682281131a4297e1daddc01740bf52e73ecd5bce2b68d2a19c758696b2d\Files\Windows\System32\DriverStore\en-US\c_securitydevices.inf_loc \\?\C:\ProgramData\Docker\windowsfilter\bd5efe6d686ac304121708322fae29028b723ac336711494139468fd00e00a81\Files\Windows\WinSxS\amd64_c_securitydevices.inf.resources_31bf3856ad364e35_10.0.17763.1_en-us_8ae2ad4f85596716\c_securitydevices.inf_loc: An attempt was made to create more links on a file than the file system supports.

After this error the Docker installation is stuck.

A look in C:\ProgramData\Docker\windowsfilter shows the problem: There are dozens of directories, each of which is up to several GB in size, so that the total amounts to 100-80 GB. There is still ~700GB of free space on the NTFS file system.

They are apparently not deleted by Docker, not even by docker system prune, which explains the error message. The directories are so write-protected that they cannot be deleted even with admin rights and Powershell scripts.

Even uninstalling Docker does not delete the folders.

However, the de/reinstallation seems to be the only way to get Docker up and running again (for two weeks).

Reproduce

docker system prune --all --force --volumes docker pull X

Expected behavior

docker system prune --all --force --volumes should cleanup C:\ProgramData\Docker\windowsfilter

docker version

Client:
 Cloud integration: v1.0.35+desktop.5
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:32:48 2023
 OS/Arch:           windows/amd64
 Context:           default

Server: Docker Desktop 4.25.0 (126437)
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.24)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:31:39 2023
  OS/Arch:          windows/amd64
  Experimental:     false

docker info

Client:
 Version:    24.0.6
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2-desktop.5
    Path:     C:\Program Files\Docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.23.0-desktop.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-compose.exe
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-dev.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.20
    Path:     C:\Program Files\Docker\cli-plugins\docker-extension.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.9
    Path:     C:\Program Files\Docker\cli-plugins\docker-init.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sbom.exe
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-scan.exe
  scout: Docker Scout (Docker Inc.)
    Version:  v1.0.9
    Path:     C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 24.0.6
 Storage Driver: windowsfilter
  Windows:
 Logging Driver: json-file
 Plugins:
  Volume: local
  Network: ics internal l2bridge l2tunnel nat null overlay private transparent
  Log: awslogs etwlogs fluentd gcplogs gelf json-file local logentries splunk syslog
 Swarm: inactive
 Default Isolation: hyperv
 Kernel Version: 10.0 22621 (22621.1.amd64fre.ni_release.220506-1250)
 Operating System: Microsoft Windows Version 22H2 (OS Build 22621.2134)
 OSType: windows
 Architecture: x86_64
 CPUs: 20
 Total Memory: 31.68GiB
 Name: a1-runner
 ID: 86c1d550-9bb9-41d1-b6e1-bfd5663861b3
 Docker Root Dir: C:\ProgramData\Docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

Diagnostics ID

0

Additional Info

No response

debug-richard commented 4 months ago

This still happens with 4.30.0.

But I found this 10 year old tool https://github.com/moby/docker-ci-zap that allows you to remove the leftovers from the file system. But you usually need to uninstall docker + reboot to close all filehandles before execution.

But the fact that this tool has 50 lines of code and the function still hasn't made it into Docker is a bit depressing.