Issue Docker Desktop, Cadvisor and WSL

AlbertoSoutullo commented 1 year ago

It looks like if you are using Windows with WSL, using cadvisor can get messy.

When you run the standard recommended cadvisor command, which is like:

sudo docker run \
  --volume=/:/rootfs:ro \
  --volume=/var/run:/var/run:ro \
  --volume=/sys:/sys:ro \
  --volume=/var/lib/docker/:/var/lib/docker:ro \
  --volume=/dev/disk/:/dev/disk:ro \
  --publish=8080:8080 \
  --detach=true \
  --name=cadvisor \
  --privileged \
  --device=/dev/kmsg \
  gcr.io/cadvisor/cadvisor:$VERSION

This might work under a linux distritubion, but this does not work under WSL.

You migth get errors like:

cadvisor | E0823 20:34:34.010493 1 manager.go:1084] Failed to create existing container: /docker/f8b0931d2a1809803906b9e25bb8285438ccabfbfc96e8a7b82cdd29d38a15d9: failed to identify the read-write layer ID for container "f8b0931d2a1809803906b9e25bb8285438ccabfbfc96e8a7b82cdd29d38a15d9". - open /var/lib/docker/image/overlay2/layerdb/mounts/f8b0931d2a1809803906b9e25bb8285438ccabfbfc96e8a7b82cdd29d38a15d9/mount-id: no such file or directory

This error is logged in the cadvisor container. You can still go to localhost:8080 and check that cadvisor looks like it is working, but it cannot show data from the docker containers, since it cannot access to it.

This issue is well explained here, but under my personal case it was a bit different.

As explained in the posted link, cadvisor is expecting to read information under /var/lib/docker. In WSL, this folder is empty. The information that cadvisor is looking for is under \\wsl$\docker-desktop-data\version-pack-data\community\docker which you can access pasting this path in File Explorer. In my personal case, don't know exactly why, it was located in \\wsl$\docker-desktop-data\data\docker. So the only steps you need to do inside WSL are:

sudo mkdir /mnt/windows_docker (or whatever folder/path you want)
sudo mount -t drvfs '\\wsl$\docker-desktop-data\version-pack-data\community\docker' /mnt/windows_docker or in my case
sudo mount -t drvfs '\\wsl$\docker-desktop-data\data\docker' /mnt/windows_docker

And then modify the cadvisor docker run command accordingly with this new information:

docker run \
--volume=/:/rootfs:ro \
--volume=/var/run:/var/run:rw \
--volume=/var/lib/docker/:/var/lib/docker:ro \
--volume=/dev/disk/:/dev/disk:ro \
--volume=/sys:/sys:ro \
--volume=/etc/machine-id:/etc/machine-id:ro \
--volume=/mnt/windows_docker/:/rootfs/var/lib/docker:ro \ # <--- new volume
--publish=8080:8080 \
--detach=true \
--name=cadvisor \
--privileged \
--device=/dev/kmsg \
gcr.io/cadvisor/cadvisor

The error should not appear now, and you will be able to access to the containers information.

Extra information:

Docker Desktop can get crazy sometimes. Engine might not start, or commands that should work suddently they are not working anymore. After extensive usage with lots of containers, docker desktop and kurtosis, I find that sometimes when something is weirdly behaving, it is good to restart the computer (sigh).

It is recommented to delete anything docker related in windows and WSL before trying this steps.

Edit: Well, it looks like the behaviour can still be erratic as hell, one day working fine and the next one it is not. Another solution I found is to mount \\wsl$\docker-desktop-data\data\docker in the home directory in WSL, something like: sudo mount -t drvfs '\\wsl$\docker-desktop-data\data\docker' /home/your_user/windows_docker God knows why, but this works much more consistently.

ofsaleem commented 1 year ago

thanks for this guide, one thing i'm not understanding is why we bind mount the new volume path to /rootfs/var/lib/docker instead of /var/lib/docker

AlbertoSoutullo commented 1 year ago

thanks for this guide, one thing i'm not understanding is why we bind mount the new volume path to /rootfs/var/lib/docker instead of /var/lib/docker

It's been half a year from this so I don't remember exactly. I don't know if it works doing it like that, if it does maybe I just didn't realize about that when I was working with this. Still, glad it was helpful for you!

kevincam3 commented 2 months ago

I thought this problem still existed because it wasn't showing anything but after few mintues while I was looking for a solution and came across this, it started working so it looks like this is no longer a problem. It just took a few minutes before it started tracking

JbmOnGitHub commented 4 weeks ago

Config : Docker Desktop 4.35.1 (173168) on Windows 10.0.19045

In my case \\wsl$\docker-desktop-data\data\docker doesn't exist But i have : \\wsl.localhost\docker-desktop\mnt\docker-desktop-disk\data\docker

But advisor container show this log : open /rootfs/var/lib/docker/image/overlayfs/layerdb/mounts/35.../mount-id no such file or directory

And it's true, \\wsl.localhost\docker-desktop\mnt\docker-desktop-disk\data\docker doesn't have any image/overlayfs/layerdb path

I don't have any idea to resolve this...

hexley21 commented 1 week ago

I don't have any idea to resolve this...

Have you found any solutions? I switched from linux to windows and i have the same issue as you, i don't even have docker-desktop-data directory

hexley21 commented 4 days ago

I don't have any idea to resolve this...

Have you found any solutions? I switched from linux to windows and i have the same issue as you, i don't even have docker-desktop-data directory

I fixed this by just running docker in WSL ubuntu, i completely deleted docker desktop from windows and all the things is done through the WSL ubuntu, everything is fine, you can hook up portainer just in case if you want UI

vacp2p / wakurtosis

Issue Docker Desktop, Cadvisor and WSL #58