[podman] Can I use kind if I change peripherals and connected devices?

[plluksie]

Hello, I encountered error which seems to be related to the fact that I often change docking station and peripherals. It seems that if I'll deploy cluster being connected to one docking station (and set of peripherals): KIND_EXPERIMENTAL_PROVIDER=podman kind create cluster --config=/home/lsiemiradzki/opt/kind/kind-cluster.yaml Then I turn it off, move to other docking station (with different set of peripherals) and try starting the main container, I'm getting: Error: unable to start container "containerID": error stat'ing file /dev/bus/usb/003/...

The workaround I'm using is to throw away the control plane container and re-create the cluster. Thanks for any hints.

I checked Known Issues and [Issues] (https://github.com/kubernetes-sigs/kind/issues?q=is%3Aissue+is%3Aopen+%22Error%3A+unable+to+start+container%22), but I haven't found anything similar.

More details:

❯ kind --version && podman -v ; podman ps -a | grep kindest   
kind version 0.14.0
podman version 3.4.2
d1b296720301  docker.io/kindest/node@sha256:0866296e693efe1fed79d5e6c7af8df71fc73ae45e3679af05342239cdc5bc8e               
         46 hours ago  Created                  0.0.0.0:8080->80/tcp, 0.0.0.0:8443->443/tcp, 127.0.0.1:36189->6443/tcp  kin
d-control-plane

☸ kind-kind in ~ 
❯ podman start d1b296720301 
Error: unable to start container "d1b29672030149cd0829fc11bfe5a1b51340cf530d00a72f112640e21bb3bd69": error stat'ing file `/
dev/bus/usb/003/007`: No such file or directory: OCI runtime attempted to invoke a command that was not found

[BenTheElder]

Sorry I've been on vacation -- This is a podman bug AFAICT, kind is not responsible for mounting devices like /dev/bus/...

[BenTheElder]

In general restart support is much better under docker, in the next release it should be pretty robust. This particular issue seems like a bug in podman though, we're not explicitly requesting this path, podman should probably gracefully handle this when mounting devices on restart.

[wherka-ama]

@BenTheElder : you are right - this has nothing to do with kind per se. In fact it goes even deeper than podman. I had a little go at it and it looks like it's crun which is actually performing all these checks. podman simply delegates it to conmon, which in turn executes crun with provided arguments(log attached).

kind-podman-conmon-crun.err.log

Good news: there is a quick work around which allows to keep the cluster intact and still restart it, even if some devices are missing. It's as simple as touching the files in question(these flagged in the console), starting the cluster and removing them again to ensure there is no problem when the actual devices are plugged in.

podman start kind-control-plane
#[ERROR]
sudo touch /dev/bus/usb/003/007 /dev/bus/usb/003/010 ..
podman start kind-control-plane
sudo rm /dev/bus/usb/003/007 /dev/bus/usb/003/010 .. 

It may not be 100% reliable as there will be cases where the container may actually need to interact with the device in the scope, but it works for me as I often plug/unplug the docking station and when I work without it.

Kind should not really be used as a production-grade cluster, so recreating it from scratch should not be a big deal. However, in some cases people may be in the middle of something and loosing quite a chunk of time for recreating it from scratch, which is not great. Making it work - even with the workaround above may be quite nice.

I know you've heard that million times before - @BenTheElder you are the guy who literally rewrote the rules for development environments for people working with Kubernetes! Kind is such a game changer I just cannot thank you enough on behalf of countless developers who used to struggle with massive feedback loops on the CI environments when testing the changes on K8s. I know there are alternatives these days, but Kind is still considered as a swiss army knife in my toolbox! Thank you!

[BenTheElder]

Sorry I've still been a bit overloaded returning from vacation and switching roles 😅

Thank you!

That's an interesting workaround, I think you coulud also find where podman is storing the pod spec and alter the mounts, but this feels like something that should be reported back to podman or crun (I can't remember which one decides the device list initially but I think it's actually podman?) as a bug, I'm pretty sure docker will not fail to start the container in this case.