Using Windows containers: json field "data-root" does not work, symlinking C:\ProgramData\Docker does not work

cbckly commented 4 months ago

Description

Have a need for Windows Containers under Windows 10. Can't use C drive, must use D drive.

Can install and start Docker Desktop, switch to Windows Containers, and watched C drive fill up after running a few dockers.
Quit Docker Desktop with the tray icon, edited daemon.json to add "data-root", escaped the slashes, made sure new path existed, and relaunched Docker Desktop. It fails to start, I believe it just times out after 30s.

[2024-02-24T17:06:03.854222100Z][com.docker.backend.exe.ipc][I] (26562eaa) 0f8911be-BackendAPI S->C d07bcc14-engines   (500µs): OK
[2024-02-24T17:06:03.854222100Z][com.docker.backend.exe.ipc][I] (26562eaa) d07bcc14-engines C<-S 0f8911be-BackendAPI GET /engine/windows/daemon.json (500µs): {"data-root":"D:\\docker\\data-windows","experimental":false}
[2024-02-24T17:06:03.854222100Z][com.docker.backend.exe.ipc][I] (a73e5c07) 881fe8ef-engines C->S com.docker.service POST /windowscontainers/start: {"daemonJSON":"{\"data-root\":\"D:\\\\docker\\\\data-windows\",\"experimental\":false}","overrideProxyExclude":"","overrideWindowsDockerdPort":"float64","userCertsDir":"\u003cHOME\u003e\\.docker\\certs.d"}
[2024-02-24T17:06:34.058765100Z][com.docker.backend.exe.ipc][W] (a73e5c07) 881fe8ef-engines C<-S 7fe7fdac-ServiceAPI POST /windowscontainers/start (30.204543s): status code not OK but 500: Unhandled exception: Time out has expired and the operation has not been completed.
[2024-02-24T17:06:34.058765100Z][com.docker.backend.exe.events][I] adding server timestamp to event (engines): 1708794394058765100 (failed to start Windows Containers engine) 
[2024-02-24T17:06:34.058765100Z][com.docker.backend.exe.engines][W] aborting ping engine: context canceled
[2024-02-24T17:06:34.058765100Z][com.docker.backend.exe.engines][E] running unix engine: starting windows containers engine: starting windows containers: status code not OK but 500: Unhandled exception: Time out has expired and the operation has not been completed.

If I launch dockerd like this, it starts error free:

"C:\Program Files\Docker\Docker\resources\dockerd.exe" -G docker-users --config-file c:\programdata\docker\config\daemon.json

But Desktop doesn't work so well doing this.... And running docker commands seems to expect a different pipe but didn't write down what it said.

I've tried setting data-root with existing installs, new installs, half dozen times trying to find the nuance and nothing worked.

Googling showed most people just set a symlink and didn't bother with data-root. Resisted this at first since it was configurable, but after hours of wasted effort, decided it was best as Docker would still create the daemon.json file on C and a bunch of other stuff anyways. Uninstalled Docker Desktop, set the symlink:

C:\ProgramData>dir /al

 Directory of C:\ProgramData

2024-02-24  02:14 PM    <JUNCTION>     Docker [D:\Docker\ProgramData-Windows]

Installed Docker Desktop, and to my utter dismay received this error message:

"Something went wrong. Starting Windows containers: status code not ok but 500. Path contains symlink: C:\ProgramData\Docker"

I can't tell if it's stopping BECAUSE of the symlink, or it's just info, but man, I wasn't expecting this to fail either. It let me create a file in C:\ProgramData\Docker okay, but otherwise there is nothing there.

Deleted the symlink and launched it again and it started up fine. Right back on C drive...

Windows Containers can not work anywhere but drive C.

(Out of pure desperation I robocopy'ed the files to D: and put the symlink in place, as expected, that also failed with the "Path contains symlink" error)

Reproduce

1: mklink /j "C:\ProgramData\Docker" "D:\Docker\ProgramData-Windows" 2: Install Docker Desktop

Launch Docker Desktop
Switch to Windows Containers if it even runs...

Expected behavior

Windows Container data should go where I designate it. Not where you want it. I couldn't even choose where Desktop installed to...

docker version

error during connect: this error may indicate that the docker daemon is not running: Get "http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.24/version": open //./pipe/docker_engine: The system cannot find the file specified.
Client:
 Cloud integration: v1.0.35+desktop.10
 Version:           25.0.3
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        4debf41
 Built:             Tue Feb  6 21:13:02 2024
 OS/Arch:           windows/amd64
 Context:           default

docker info

Client:
 Version:    25.0.3
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1-desktop.4
    Path:     C:\Program Files\Docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.5-desktop.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-compose.exe
  debug: Get a shell into any image or container. (Docker Inc.)
    Version:  0.0.24
    Path:     C:\Program Files\Docker\cli-plugins\docker-debug.exe
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-dev.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.21
    Path:     C:\Program Files\Docker\cli-plugins\docker-extension.exe
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     C:\Program Files\Docker\cli-plugins\docker-feedback.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.0.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-init.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sbom.exe
  scout: Docker Scout (Docker Inc.)
    Version:  v1.4.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
ERROR: error during connect: this error may indicate that the docker daemon is not running: Get "http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.24/info": open //./pipe/docker_engine: The system cannot find the file specified.
errors pretty printing info

Diagnostics ID

F64681A5-50B9-4EC7-A70F-76EC829A85DD/20240224201239

Additional Info

No response

alubensky-schubert commented 2 months ago

Same problem here. There are so many people having the same problem, many claim one of the solutions work, but nope, they don't (anymore?).

KeganHollern commented 1 week ago

I found the issue was with the new data-root folder's permissions.

The daemon service attempts to set permissions to your new data-root folder, but there must be a bug in this process:

[2024-06-24T19:38:13.692148100Z][WindowsContainersController][I] data-root key found in options, applying permissions rules on D:\docker\data-root
[2024-06-24T19:38:13.696165400Z][WindowsContainersController][W] D:\docker\data-root: Attempted to perform an unauthorized operation.
[2024-06-24T19:38:13.696717400Z][WindowsContainersController][W] D:\docker\data-root: Access to the path 'D:\docker\data-root' is denied.
[2024-06-24T19:38:13.696717400Z][WindowsContainersController][W] D:\docker\data-root: Access to the path 'D:\docker\data-root' is denied.

This ends up corrupting the folder permissions, so further down the line,

[2024-06-24T19:38:13.873601000Z][WindowsDaemon][E] fatal: open D:\docker\data-root\panic.log: Access is denied.

The Fix

When setting the permissions to Exactly Match C:\ProgramData\Docker the service did not apply permissions:

[2024-06-24T19:40:21.084984200Z][WindowsContainersController][I] data-root key found in options, applying permissions rules on D:\docker\data-root

And the daemon starts successfully.

Extra Notes

My data-root folder was empty - I did not attempt to migrate anything from C:\ProgramData\Docker.

I manually modified my daemon-windows.json in my User Home directory while Docker Desktop was Shut Off.

I used https://github.com/moby/docker-ci-zap to delete my C:\ProgramData\Docker folder before attempting to start Docker Desktop.

Exact Permissions were like:

Folder owner: "Administrators"
Full Control for SYSTEM and Administrators
Disable permission inheritance but, please refer to the permissions of your own docker data folder on C drive.

Good luck to anyone reading this post !

docker / for-win