Open ykis-0-0 opened 1 year ago
nzbr
commented
1 year ago The docker desktop proxy's cli unfortunately doesn't seem to be stable... This is the second time it broke due to a change to the CLI. It'd be much better if we could get docker-desktop itself to start the proxy like it does on other distros, but I couldn't even get docker desktop itself to work last time I tried using it
I'll look into it further when I have the time (Likely next week)
nzbr
commented
1 year ago I tried to debug it, but I can't get Docker Desktop to work at all right now. It just keeps crashing every few minutes
ykis-0-0
commented
1 year ago That seems strange, the Docker Desktop in my machine works pretty normal
OTOH, is it able to inject some $PATHs into the shell called by wsl.exe -d NixOS? I think that may solve the issue.
ykis-0-0
commented
1 year ago Supplementary info: bash didn't even try 😂
> wsl.exe -d NixOS echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib
> wsl.exe -d NixOS echo $-
hBc
> wsl.exe -d NixOS shopt login_shell
login_shell offThe weird thing to me is, that Docker Desktop says that wsl.exe -d NixOS uname fails, but I can run that just fine and it returns Linux like it should. But then again: The docker engine just crashes after a few minutes, even when I try to interact with it from a Windows shell
ykis-0-0
commented
1 year ago The weird thing to me is, that Docker Desktop says that
wsl.exe -d NixOS unamefails, but I can run that just fine and it returnsLinuxlike it should.
I wonder may that be related to the syschemd / native systemd configs. Maybe it's the native one causing some problems?
But then again: The docker engine just crashes after a few minutes, even when I try to interact with it from a Windows shell
That's really strange, did you try to set another distro as default and see if the crashing persists? Maybe that even isn't NixOS' problem.
LGUG2Z
commented
1 year ago Is there any idea which version of Docker resulted in these breaking changes? I wonder if we could temporarily work around this by pinning the version of Docker Desktop installed using winget or something similar.
ykis-0-0
commented
1 year ago Is there any idea which version of Docker resulted in these breaking changes?
Sorry, but I've lost track of the Docker version at the time, since I had hadn't spin up Docker Desktop when I found this issue. I'm willing to do a bisect, but where could I find the old versions?
The weird thing to me is, that Docker Desktop says that
wsl.exe -d NixOS unamefails, but I can run that just fine and it returnsLinuxlike it should.
On a second thought, it maybe a misconfiguration of mine. For a comparison, May I ask the output of wsl.exe -d NixOS echo $PATH on your machine? Maybe my WSL calls the wrong bash I wonder
nzbr
commented
1 year ago Docker Desktop stops crashing when I set my default distro to Ubuntu, so it's definitely a problem with NixOS and not Docker Desktop
packruler
commented
1 year ago Docker Desktop stops crashing when I set my default distro to Ubuntu, so it's definitely a problem with NixOS and not Docker Desktop
I am seeing the same cause and effect. I have reinstalled Docker Desktop after it hangs on startup. I have tried enabling NixOS integration via Docker Desktop WSL settings as well as making NixOS my default after Docker Desktop install.
I think it is safe to say there is something in the NixOS Docker Desktop connection but I am unaware of methods to debug further.
Vinetos
commented
1 year ago I used https://github.com/a2o/snoopy to monitor the commands Docker Desktop does to WSL instances on the first run and here is the output for Ubuntu.
In all my test, Ubuntu was the default WSL2.
As we can see on the last line, Docker Desktop is mounting a Windows folder to /Docker/host in the WSL.
/sbin/mount.drvfs C:\Program Files\Docker\Docker\resources /Docker/host -o rw,noatimeAfter trying some dumb arguments for the docker-desktop-user-proxy, It seems that the CLI wants the Windows path of resources folder which will be mounted to /Docker/host . I can successfully launch the proxy manually with sudo /mnt/wsl/docker-desktop/docker-desktop-user-distro proxy "C:\Program Files\Docker\Docker\resources".
But docker ps does not work on another terminal. (See why in UPDATE 2)
[vinetos@wsl:~]$ sudo docker ps
fork/exec /mnt/wsl/docker-desktop/cli-tools/usr/bin/com.docker.cli: no such file or directory
[vinetos@wsl:~]$ /mnt/wsl/docker-desktop/cli-tools/usr/bin/com.docker.cli
-bash: /mnt/wsl/docker-desktop/cli-tools/usr/bin/com.docker.cli: cannot execute: required file not foundProbably a missing file or something like that. (See why in UPDATE 2)
Also, running /mnt/c/Program\ Files/Docker/Docker/resources/com.docker.diagnose.exe check give me :
`[FAIL] DD0014: are the backend processes running? 1 error occurred:
My Investigation continue.
UPDATE 1:
After updating the systemd script to add the folder C:\Program Files\Docker\Docker\resources, I can reproduce the same behavior as running the command manually :
systemd.services.docker-desktop-proxy = {
description = "Docker Desktop proxy";
path = [ pkgs.mount ];
script = ''
${config.wsl.wslConf.automount.root}/wsl/docker-desktop/docker-desktop-user-distro proxy --docker-desktop-root ${config.wsl.wslConf.automount.root}/wsl/docker-desktop "C:\Program Files\Docker\Docker\resources"
'';
wantedBy = [ "multi-user.target" ];
serviceConfig = {
Restart = "on-failure";
RestartSec = "30s";
};
};
users.groups.docker.members = [
config.wsl.defaultUser
];
};But docker ps still not working because it calls the executable of docker-desktop (the WSL distribution).
UPDATE 2: A work around is the use the host command-line tools to run the docker command successfully.
I can use /Docker/host/bin/com.docker.cli.exe run hello-world successfully but /Docker/host/bin/docker does not works still (see UPDATED 3).
Using alias docker=com.docker.cli.exe in the terminal allow me to run all docker commands by passing through Windows.
Docker Desktop still fails when using NixOS as default WSL2 probably because the weird alias tricks.
UPDATE 3:
The script located in C:\Program Files\Docker\Docker\resources\bin\docker (mounted in /Docker/host/bin/docker) is used to pass Windows docker command to WSL. This is why the "host" docker command is not working. Here is the script :
#!/usr/bin/env sh
#
# Copyright (c) Docker Inc.
binary=$(basename "$0")
case "$(uname -r)" in
*microsoft* )
# WSL 2
if [ -f "/usr/bin/$binary" ]; then
# It seems this script ran before the integration added the real binary.
# Run the real binary for the user.
set -e
"/usr/bin/$binary" "$@" # <============== here
else
echo
echo "The command '$binary' could not be found in this WSL 2 distro."
echo "We recommend to activate the WSL integration in Docker Desktop settings."
echo
echo "For details about using Docker Desktop with WSL 2, visit:"
echo
echo "https://docs.docker.com/go/wsl2/"
echo
exit 1
fi
;;
*Microsoft* )
# WSL 1
echo
echo "The command '$binary' could not be found in this WSL 1 distro."
echo "We recommend to convert this distro to WSL 2 and activate"
echo "the WSL integration in Docker Desktop settings."
echo
echo "For details about using Docker Desktop with WSL 2, visit:"
echo
echo "https://docs.docker.com/go/wsl2/"
echo
exit 1
;;
* )
# No WSL, maybe MINGW, try Windows exe which should also be in PATH
"$binary.exe" "$@"
;;
esacThe line "/usr/bin/$binary" will execute/usr/bin/docker which will fail.
[vinetos@wsl:~]$ ls -sla /usr/bin/ | grep docker
0 lrwxrwxrwx 1 root root 56 Jul 30 16:22 com.docker.cli -> /mnt/wsl/docker-desktop/cli-tools/usr/bin/com.docker.cli
0 lrwxrwxrwx 1 root root 48 Jul 30 16:22 docker -> /mnt/wsl/docker-desktop/cli-tools/usr/bin/docker
0 lrwxrwxrwx 1 root root 56 Jul 30 16:22 docker-compose -> /mnt/wsl/docker-desktop/cli-tools/usr/bin/docker-compose
0 lrwxrwxrwx 1 root root 59 Jul 30 16:22 docker-compose-v1 -> /mnt/wsl/docker-desktop/cli-tools/usr/bin/docker-compose-v1
0 lrwxrwxrwx 1 root root 46 Jul 30 16:22 docker-credential-desktop.exe -> /Docker/host/bin/docker-credential-desktop.exe
0 lrwxrwxrwx 1 root root 54 Jul 30 16:22 docker-index -> /mnt/wsl/docker-desktop/cli-tools/usr/bin/docker-index
0 lrwxrwxrwx 1 root root 50 Jul 30 16:22 hub-tool -> /mnt/wsl/docker-desktop/cli-tools/usr/bin/hub-toolSo, executing the docker bin from docker-desktop in the NixOS WSL fails because of "missing file" (maybe PATH + LIB issue).
Using the Windows host to pass docker commands (and bypassing the docker script) can be a temporary fix.
Another solution is to use the nix-packaged docker client. By running /nix/store/<HASH>-system-path/bin/docker ps, the Docker client works fine too. But, It will not make the other CLI tools (hub-tool, docker-index) available but the docker command will be working if we override these paths. And that's could be why Docker Desktop fails.
What do you think ?
kfish610
commented
1 year ago For the record, when I run the proxy exe on my machine (also nativeSystemd, Docker v4.21.1), it doesn't even get past the first WSL command. The output is
PS C:\Users\kfish> & 'C:\Program Files\Docker\Docker\resources\com.docker.wsl-distro-proxy.exe' --distro-name NixOS
[2023-07-31T02:49:35.712300000Z][com.docker.wsl-distro-proxy.exe.paths][I] set path configuration to OnHost
[2023-07-31T02:49:35.712300000Z][com.docker.wsl-distro-proxy.exe][I] start wsl integration for distro NixOS
[2023-07-31T02:49:35.712818000Z][com.docker.wsl-distro-proxy.exe.linuxworkspaces][I] wsl.exe -d NixOS -e sh -c mkdir -p ~/.docker/run
[2023-07-31T02:49:35.853652800Z][com.docker.wsl-distro-proxy.exe][F] unable to create ~/.docker/run directory: exit status 127
[main.main()
[ win/backend/cmd/com.docker.wsl-distro-proxy/main.go:138 +0xe4bIndeed, when I run wsl.exe -d NixOS -e sh -c mkdir -p ~/.docker/run in Powershell, I do get an exit code 127 (-p: line 1: mkdir: command not found), though I do not get the same exit code if I run sh -c mkdir -p ~/.docker/run in WSL; I get exit code 1 (mkdir: missing operand). Not sure what's up with that.
Vinetos
commented
1 year ago Indeed, when I run
wsl.exe -d NixOS -e sh -c mkdir -p ~/.docker/runin Powershell, I do get an exit code 127 (-p: line 1: mkdir: command not found), though I do not get the same exit code if I runsh -c mkdir -p ~/.docker/runin WSL; I get exit code 1 (mkdir: missing operand). Not sure what's up with that.
What version are you running ?
Before my tests, I have updated WSL2 (wsl --update) and used the last version of the installer (23.05 on master here) and switch to unstable inputs (my configuration : https://github.com/Vinetos/nixos ).
Does it run if you log into the WSL ? I have only tested with the service and by opening a shell. Maybe it's another king of bug that make impossible to run command without dropping into a shell.
EDIT: This is linked to https://github.com/nix-community/NixOS-WSL/issues/222 because Windows skips the shell wrapper
daspk04
commented
6 months ago Hi,
I just installed a fresh NixOS-WSL, when I enable Docker Desktop in /etc/nixos/configuration.nix, I get this error,
Loaded: loaded (/etc/systemd/system/docker-desktop-proxy.service; enabled; preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Sun 2024-01-28 12:57:04 UTC; 286ms ago
Process: 3655 ExecStart=/nix/store/qzqkfrkd9hh0f80zb9sb1hw79bhfm30j-unit-script-docker-desktop-proxy-start/bin/docker-desktop-proxy-start (code=exited, status=1/FAILURE)
Main PID: 3655 (code=exited, status=1/FAILURE)
IP: 0B in, 0B out
CPU: 18ms
warning: error(s) occurred while switching to the new configurationManually running the script fails as well.
Command:
sudo /nix/store/qzqkfrkd9hh0f80zb9sb1hw79bhfm30j-unit-script-docker-desktop-proxy-start/bin/docker-desktop-proxy-startError:
Error: accepts 1 arg(s), received 0
[2024-01-28T13:05:01.670402107Z][docker-desktop-user-distro][F] accepts 1 arg(s), received 0
[main.main()
[ linuxkit/pkg/docker-desktop-user-distro/main.go:594 +0x50I assume a path as an argument is missing from the script as mentioned here ?
Any suggestion ?
RyzeNGrind
commented
6 months ago I tried to install NixOS-WSL2 and integrate with Docker Desktop on W10 and I am receiving similar errors. Is there any workaround or fix for this to get nixos integrated with docker desktop and kubernetes?
daspk04
commented
6 months ago I tried to install NixOS-WSL2 and integrate with Docker Desktop on W10 and I am receiving similar errors. Is there any workaround or fix for this to get nixos integrated with docker desktop and kubernetes?
Hi @RyzeNGrind,
I tried to follow as suggested here. This seems to work for me.
Basically this is my configuration for wsl:
wsl = {
enable = true;
wslConf.automount.root = "/mnt";
wslConf.interop.appendWindowsPath = false;
wslConf.network.generateHosts = false;
defaultUser = username;
startMenuLaunchers = true;
# Enable integration with Docker Desktop (needs to be installed)
docker-desktop.enable = false;
extraBin = with pkgs; [
# Binaries for Docker Desktop wsl-distro-proxy
{ src = "${coreutils}/bin/mkdir"; }
{ src = "${coreutils}/bin/cat"; }
{ src = "${coreutils}/bin/whoami"; }
{ src = "${coreutils}/bin/ls"; }
{ src = "${busybox}/bin/addgroup"; }
{ src = "${su}/bin/groupadd"; }
{ src = "${su}/bin/usermod"; }
];
};
virtualisation.docker = {
enable = true;
enableOnBoot = true;
autoPrune.enable = true;
};
## patch the script
systemd.services.docker-desktop-proxy.script = lib.mkForce ''${config.wsl.wslConf.automount.root}/wsl/docker-desktop/docker-desktop-user-distro proxy --docker-desktop-root ${config.wsl.wslConf.automount.root}/wsl/docker-desktop "C:\Program Files\Docker\Docker\resources"'';docker-desktop.enable = false from NixOS but I enabled it from the host (i.e. windows) Docker Desktop.❯ which -a docker
/usr/bin/docker
/run/current-system/sw/bin/dockerPossible because of this, as Docker Desktop on host (windows) expects no docker to be installed inside the distro and the official document suggests it to be removed and given that NixOS distro already has a docker installed possibly that creates a conflict:
/run/current-system/sw/bin/docker is the one that works fine and shows the container and images that I already have on my host system./run/current-system/sw/bin/docker info
or
sudo /run/current-system/sw/bin/docker info
## Docker info
Client:
Version: 24.0.5
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.12.0-desktop.2
Path: /usr/local/lib/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.23.3-desktop.2
Path: /usr/local/lib/docker/cli-plugins/docker-compose
dev: Docker Dev Environments (Docker Inc.)
Version: v0.1.0
Path: /usr/local/lib/docker/cli-plugins/docker-dev
extension: Manages Docker extensions (Docker Inc.)
Version: v0.2.21
Path: /usr/local/lib/docker/cli-plugins/docker-extension
feedback: Provide feedback, right in your terminal! (Docker Inc.)
Version: 0.1
Path: /usr/local/lib/docker/cli-plugins/docker-feedback
init: Creates Docker-related starter files for your project (Docker Inc.)
Version: v0.1.0-beta.10
Path: /usr/local/lib/docker/cli-plugins/docker-init
sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
Version: 0.6.0
Path: /usr/local/lib/docker/cli-plugins/docker-sbom
scan: Docker Scan (Docker Inc.)
Version: v0.26.0
Path: /usr/local/lib/docker/cli-plugins/docker-scan
scout: Docker Scout (Docker Inc.)
Version: v1.2.0
Path: /usr/local/lib/docker/cli-plugins/docker-scout
Server:
Containers: 9
Running: 1
Paused: 0
Stopped: 8
Images: 11
Server Version: 24.0.7
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
runc version: v1.1.10-0-g18a0cb0
init version: de40ad0
Security Options:
seccomp
Profile: unconfined
cgroupns
Kernel Version: 5.15.133.1-microsoft-standard-WSL2
Operating System: Docker Desktop <----------------------- Check this
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 27.41GiB
Name: docker-desktop <------------------------------------- Check this
ID: ce4faac5-bea5-4a89-88a4-bbc39da52da6
Docker Root Dir: /var/lib/docker
Debug Mode: false
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
No Proxy: hubproxy.docker.internal
Experimental: false
Insecure Registries:
hubproxy.docker.internal:5555
127.0.0.0/8
Live Restore Enabled: false
WARNING: daemon is not using the default seccomp profiledocker info for Server points to docker desktop distro which is what is correct at least in my understanding as mentioned in Docker official Doc
Operating System: Docker Desktop
Name: docker-desktop/run/current-system/sw/bin/docker as docker for the shell in nix configuration. docker = "/run/current-system/sw/bin/docker";Summary:
docker-desktop.enable = false; extra bin in configuration.nixNixOS distro from Docker Desktop on host (windows)/run/current-system/sw/bin/docker
schlich
commented
5 months ago Brilliant solution! I was banging my head on the wall for hours. god bless this voodoo magic.
Mach1212
commented
4 months ago Thanks for the solution! All I had to do was add the wsl.extrabin option for my Docker Desktop to work.
lyra95
commented
3 weeks ago The docker-desktop-user-distro proxy command line api (this program is on windows, so cannot be managed by NixOS) keeps changing as one updates docker desktop version.
So I decide to override the systemd configuration here: https://github.com/nix-community/NixOS-WSL/blob/f373ad59ae5866f0f98216bd5c71526b373450d2/modules/docker-desktop.nix#L23-L33
I made it as a module:
# /etc/nixos/modules/docker-desktop-fix.nix
{
config,
lib,
pkgs,
...
}:
with lib;
{
options.fix.docker-desktop.enable = mkEnableOption "docker desktop fix";
config =
let
resources = "C:\\Program Files\\Docker\\Docker\\resources";
in
mkIf (config.wsl.docker-desktop.enable && config.fix.docker-desktop.enable) {
systemd.services.docker-desktop-proxy = {
script = mkForce ''
${config.wsl.wslConf.automount.root}/wsl/docker-desktop/docker-desktop-user-distro proxy --docker-desktop-root ${config.wsl.wslConf.automount.root}/wsl/docker-desktop '${resources}'
'';
path = [ pkgs.mount ];
};
};
}To apply the override, simply import it in your configuration.nix file and enable it.
# /etc/nixos/configuration.nix
{
config,
lib,
pkgs,
...
}:
{
imports = [
# ...
./modules/docker-desktop-fix.nix
];
# ...
wsl.docker-desktop.enable = true;
fix.docker-desktop.enable = true;
}
Bug description
(Probably) outdated docker-desktop-proxy scripts prevented the integration from running.
My environment:
(核心版本 means kernel version, i guess)
nixpkgs: NixOS/nixpkgs@e3652e0NixOS-WSL: nix-community/NixOS-WSL@64558a3My guess on the situation:
/mnt/wsl/docker-desktop/docker-desktop-user-distronow expects a mandatory argument of the socket fileuname, which doesn't exists in$PATHwhen called bywsl.exe -d NixOS(which itself is expected cause it's NixOS)To Reproduce
Steps to reproduce the behavior:
config.wsl.docker-desktop.enabletotruesystemd.services.docker-desktop-proxywith this: (To solve problem 1)sudo nixos-rebuild test --flake .Expected behavior
The
docker-desktop-proxy.serviceshould load successfully onnixos-rebuildLogs