kubectl works, but can't get docker or nerdctl to work

foobarbecue commented 2 years ago

Rancher Desktop Version

1.0.1

Rancher Desktop K8s Version

v1.22.6

Which container runtime are you using?

moby (docker cli)

What operating system are you using?

Windows

Operating System / Build Version

Windows 10 Enterprise 19042.1466

What CPU architecture are you using?

x64

Linux only: what package format did you use to install Rancher Desktop?

No response

Windows User Only

No response

Actual Behavior

With a fresh Rancher Desktop install, the docker command does not work. If I switch to containerd, the nerdctl command does not work. In both cases, kubectl seems to work fine.

Steps to Reproduce

Install Rancher Desktop, selecting dockerd
open Windows Terminal and start either an ubuntu 20.04 or rancher-desktop tab
Run docker info

Result

With a fresh install of Rancher 1.0.1, kubectl works (can pull images and run pods fine), but the docker command does not:

$ docker info
Client:
 Context:    default
 Debug Mode: false

Server:
ERROR: request returned Bad Gateway for API route and version http://%2Fvar%2Frun%2Fdocker.sock/v1.24/info, check if the server supports the requested API version
errors pretty printing info

$ which docker
/mnt/c/Users/aacurtis/AppData/Local/Programs/Rancher Desktop/resources/resources/linux/bin/docker

I also tried the containerd runtime and when I do that nerdctl doesn't work:

$ nerdctl
/mnt/c/Users/aacurtis/AppData/Local/Programs/Rancher Desktop/resources/resources/linux/bin/nerdctl: 13: /mnt/c/Users/aacurtis/AppData/Local/Programs/Rancher Desktop/resources/resources/linux/bin/../lima/bin/limactl: not found
Rancher Desktop is not running. Please start Rancher Desktop to use nerdctl

Expected Behavior

I expect docker info to print a bunch of info and no errors

Additional Information

No response

ericpromislow commented 2 years ago

Did you enable the Ubuntu integration in the Rancher Desktop UI?

foobarbecue commented 2 years ago

Did you enable the Ubuntu integration in the Rancher Desktop UI?

Yes:

I have an old WSL1 install that I don't want to use for Rancher Desktop so I didn't mind that it failed. Would be nice to have the WSL2 20.04 working properly.

ericpromislow commented 2 years ago

I'm wondering if everything is running normally in the rancher-desktop distro. For example, does ps auxww | grep /usr/bin/dockerd show the command running? What rare the permissions on the two docker.sock files in the command-line?

Also, were you running Docker Desktop previously on this system? Because it's possible that there are some settings that are interfering with Rancher Desktop.

spirilis commented 2 years ago

I've confirmed this same behavior from "nerdctl" using an Alpine linux WSL integration.

EB-SURFACE:~$ nerdctl /mnt/c/Users/spiri/AppData/Local/Programs/Rancher Desktop/resources/resources/linux/bin/nerdctl: line 13: /mnt/c/Users/spiri/AppData/Local/Programs/Rancher Desktop/resources/resources/linux/bin/../lima/bin/limactl: not found Rancher Desktop is not running. Please start Rancher Desktop to use nerdctl

Jan Dubois on #rancher-desktop suggested I post an issue - just adding to this one instead. He said nerdctl can be run from the Windows shell, and that does work. Suggested the wrong version of the nerdctl script might be installed in the WSL integration.

edit:

Kubernetes version used: v1.18.20 Host O/S: Windows 11 64-bit Container runtime: containerd

mook-as commented 2 years ago

Hi! We check for WSL by looking at /proc/version (i.e. the kernel version) to see if it contains microsoft.*wsl; do either of you use a custom kernel by any chance? Or could you post the contents of /proc/version (from the failing WSL distributiuon)?

(Both of the messages are about trying to run lima, even though they're on Windows.)

Thanks!

krishnakv commented 2 years ago

Hi Folks, facing the same issue. Have installed Rancher Desktop v1.01 on Windows 10. WSL2 distro is Ubuntu 20.04.4 LTS

cat /proc/version returns the following string:

Linux version 4.4.0-19041-Microsoft (Microsoft@Microsoft.com) (gcc version 5.4.0 (GCC) ) #1237-Microsoft Sat Sep 11 14:32:00 PST 2021

Edit: for some reason, the default version on WSL install is Ver 1 , instead of 2. Have upgraded the distribution to WSl 2 and it works. Will continue testing, but looks good.

mook-as commented 2 years ago

That looks like a WSL1 kernel string; note that WSL1 isn't supported, only WSL2.

I thought we excluded WSL1 distributions from that list; we'll need to double check that.

bookmanj commented 2 years ago

Hi, I don't want to sound like a parrot, but I am having the same behavior.

Windows 10
Rancher Desktop v1.01
Ubuntu 20.04.3 LTS

cat /proc/version 
Linux version 5.10.60.1-microsoft-standard-WSL2 (oe-user@oe-host) (x86_64-msft-linux-gcc (GCC) 9.3.0, GNU ld (GNU Binutils) 2.34.0.20200220) #1 SMP Wed Aug 25 23:20:18 UTC 2021

mook-as commented 2 years ago

cat /proc/version 
Linux version 5.10.60.1-microsoft-standard-WSL2 (oe-user@oe-host) (x86_64-msft-linux-gcc (GCC) 9.3.0, GNU ld (GNU Binutils) 2.34.0.20200220) #1 SMP Wed Aug 25 23:20:18 UTC 2021
That's a WSL2 version string, and much more interesting (for me at least)! Do you have the same error in the logs (containing ../lima/bin/limactl: not found)?

bookmanj commented 2 years ago

Hey mook-as, sorry for the delay getting back to you.

I could not find any errors containing "../lima/bin/limactl: not found" in the rancher-desktop logs, but I did find the following lines in the rancher-desktop docker.log.

time="2022-02-28T13:32:45.918635000Z" level=info msg="Daemon has completed initialization"
time="2022-02-28T13:32:45.936942300Z" level=info msg="API listen on /mnt/wsl/rancher-desktop/run/docker.sock"
time="2022-02-28T13:32:45.940542700Z" level=info msg="API listen on /var/run/docker.sock"

which I thought was strange because I get "Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?" when every I run "docker ps" or "sudo docker ps".

But since the log said the API listen on "/mnt/wsl/rancher-desktop/run/docker.sock" I tried the following:

docker -H unix:///mnt/wsl/rancher-desktop/run/docker.sock ps
Got permission denied while trying to connect to the Docker daemon socket at unix:///mnt/wsl/rancher-desktop/run/docker.sock: Get "http://%2Fmnt%2Fwsl%2Francher-desktop%2Frun%2Fdocker.sock/v1.24/containers/json": dial unix /mnt/wsl/rancher-desktop/run/docker.sock: connect: permission denied

and with sudo

sudo docker -H unix:///mnt/wsl/rancher-desktop/run/docker.sock ps
CONTAINER ID   IMAGE                        COMMAND                  CREATED         STATUS         PORTS     NAMES
a81068c93190   2cd4bc25ad14                 "/entrypoint.sh --gl…"   5 minutes ago   Up 5 minutes             k8s_traefik_traefik-55fdc6d984-48xlt_kube-system_daaf8c2b-57ad-432c-8468-7e744572d7be_1
59ba0154838e   f73640fb5061                 "/metrics-server --c…"   5 minutes ago   Up 5 minutes             k8s_metrics-server_metrics-server-ff9dbcb6c-4tfqm_kube-system_b6eb21c4-5ce5-45ce-872e-fcf0293d0fcc_1
7abadf532384   fb9b574e03c3                 "local-path-provisio…"   5 minutes ago   Up 5 minutes             k8s_local-path-provisioner_local-path-provisioner-84bb864455-ptbzr_kube-system_4a07b49a-eb7f-4492-a86a-b8544648d7bb_1
94fa41dcac57   rancher/mirrored-pause:3.6   "/pause"                 5 minutes ago   Up 5 minutes             k8s_POD_metrics-server-ff9dbcb6c-4tfqm_kube-system_b6eb21c4-5ce5-45ce-872e-fcf0293d0fcc_1
c46fb7120a6d   rancher/mirrored-pause:3.6   "/pause"                 5 minutes ago   Up 5 minutes             k8s_POD_traefik-55fdc6d984-48xlt_kube-system_daaf8c2b-57ad-432c-8468-7e744572d7be_1
fccd72d16598   a4ca41631cc7                 "/coredns -conf /etc…"   5 minutes ago   Up 5 minutes             k8s_coredns_coredns-96cc4f57d-8z2v4_kube-system_c3a47a60-c680-4e89-9b3d-d552b175e307_1
e505aa4d1a8a   rancher/mirrored-pause:3.6   "/pause"                 5 minutes ago   Up 5 minutes             k8s_POD_local-path-provisioner-84bb864455-ptbzr_kube-system_4a07b49a-eb7f-4492-a86a-b8544648d7bb_1
45def73ae507   rancher/mirrored-pause:3.6   "/pause"                 5 minutes ago   Up 5 minutes             k8s_POD_coredns-96cc4f57d-8z2v4_kube-system_c3a47a60-c680-4e89-9b3d-d552b175e307_1
164f5fce9f0c   746788bcc27e                 "entry"                  5 minutes ago   Up 5 minutes             k8s_lb-port-443_svclb-traefik-fd7ln_kube-system_90566e12-652c-464c-beb5-ad3aaf0d91a4_1
2f3d31fadee4   746788bcc27e                 "entry"                  5 minutes ago   Up 5 minutes             k8s_lb-port-80_svclb-traefik-fd7ln_kube-system_90566e12-652c-464c-beb5-ad3aaf0d91a4_1
e2daef7c0d4f   rancher/mirrored-pause:3.6   "/pause"                 5 minutes ago   Up 5 minutes             k8s_POD_svclb-traefik-fd7ln_kube-system_90566e12-652c-464c-beb5-ad3aaf0d91a4_1

So I am wondering if I should chown, chmod and create a context for rancher-desktop for "unix:///mnt/wsl/rancher-desktop/run/docker.sock" or do you have a better recomendation?

thx

mook-as commented 2 years ago

@bookmanj In your case (which is getting further away from the initial report — sorry @foobarbecue) it sounds like the WSL integration is having issues. This has to do with the design of WSL — there's a separate mount namespace per distribution. So, debugging your issue involves:

Checking if the WSL integration checkbox is checked in the Rancher Desktop UI, in the WSL Integrations tab.
Checking if there's a wsl-helper docker-proxy serve process running inside your WSL distribution (which is what that checkbox does)
If yes, copy down the command it's running, stop the integration, and re-run it with --verbose=9 (as root) to see if you can see any extra details. This is a even-more-detailed version of wsl-exec-<distro>.log you'd see in the Rancher Desktop logs directory.

Any details you could provide to help us track down why it isn't working correctly would be appreciated.

bookmanj commented 2 years ago

@mook-as, thank you for the helpful information and sorry if this caused a tangent discussion, and I will try to minimize any distractions.

Checking if the WSL integration checkbox is checked in the Rancher Desktop UI, in the WSL Integrations tab. -- this is not checked and I am unable to check the box -- can I assume this is my issue and will need to resolve that further debugging

Update: I saw there was a new version of rancher desktop v1.1.0, so I uninstall v1.0.1 and install v1.1.0.

once installation was complete, I remove the /home//.kube/config and click the checkbox for WSL integration

everything is working as expected.... thx @mook-as :-)

jandubois commented 2 years ago

Looks like the issue is resolved for @krishnakv and @bookmanj.

@foobarbecue never posted the content of /proc/version, so there is nothing left to look into. Please re-open with the additional info if the problem is still reproducible in 1.2.1 or later!

rancher-sandbox / rancher-desktop