[virtualization][kubernetes_development] Troubleshooting containers

[LukeShortCloud]

Use nsenter -t <CONTAINER_PID> on the worker node a container is running on to connect into it.

Alternatively, use the alpha kubectl debug feature. It requires the EphemeralContainers features to be enabled. https://kubernetes.io/docs/tasks/debug-application-cluster/debug-running-pod/#ephemeral-container https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/

[LukeShortCloud]

The nsenter command works in a docker-shim world but that has been deprecated in newer versions of Kubernetes. I have not found a way to get this working on my Kubernetes cluster that is using containerd as the CRI.

[LukeShortCloud]

Here's the thing: nsenter actually requires a shell to work. It uses /bin/sh by default. If there is no shell installed in the container then nsenter simply won't work.

https://man7.org/linux/man-pages/man1/nsenter.1.html

I was able to get crictl working by using crictl --runtime-endpoint /run/containerd/containerd.sock. Trying to get the target PID is more complicated (read: extra steps) than it is with Docker.

# Find Pod ID.
$ sudo crictl --runtime-endpoint /run/containerd/containerd.sock pods
# Find container ID.
$ sudo crictl --runtime-endpoint /run/containerd/containerd.sock ps | grep <POD_ID>
# Enter the container.
$ crictl --runtime-endpoint /run/containerd/containerd.sock exec -it <CONTAINER_ID> /bin/sh

Alternatively, this is how nsenter is used:

# Find the process ID of the container
$ sudo crictl --runtime-endpoint /run/containerd/containerd.sock inspect 60130c0dd19db | grep pid
# Enter the container.
$ sudo nsenter -t <CONTAINER_PID>

https://github.com/towolf/kubectl-nsenter

[LukeShortCloud]

So tl;dr is that: nsenter == kubectl exec -it <POD> /bin/sh

kubectl debug helps for situations where the container does NOT have a shell

[LukeShortCloud]

Documentation has been added for containerd and crictl.

Documentation is still required for nsenter.