search

bottlerocket-os / bottlerocket

An operating system designed for hosting containers
https://bottlerocket.dev
Other
8.78k stars 519 forks source link

Question: how to use "kubectl debug node" with bottlerocket? #4011

Closed max-rocket-internet closed 4 months ago

max-rocket-internet commented 5 months ago

kubectl added some great options under the debug command and even the main Kubernetes docs have a page about debugging nodes with it here. You can see some technical details here about how it works.

kubectl debug node/ip-10-10-10-10.compute.internal -it --image=ubuntu

But obviously chroot /host does not work.

Is there some way to make this work? To access to enter-admin-container from kubectl debug node? It would be great because it would allow host debug without leaving kubectl 🙏

arnaldo2792 commented 5 months ago

There is a way (and TIL, thanks for sharing kubectl debug!). You can do this:

kubectl debug node/<node> -it --image=<image>:<tag> --profile=sysadmin

And in the newly created pod:

chroot /host apiclient exec admin bash

And you should be at the admin container now (provided that you have the admin container enabled). But, keep in mind the high privilege you are giving to the pod, make sure you its gone after you are done debugging!

But obviously chroot /host does not work.

It does! But by default it will try to execute a shell, which we don't have in Bottlerocket. So, you can still run commands like chroot /host journalctl or chroot /host ctr -n k8s.io tasks ls to debug some things, but you will be limited to what the host provides. That's why I posted how to enter the admin container from the debug pod, so that you can install tools if you need them.

max-rocket-internet commented 5 months ago

OK interesting @arnaldo2792! But where do you get this profile? I'm using versionv1.28.3 and it's not there:

~ kubectl debug node/ip-10-10-19-38.eu-west-1.compute.internal --profile=sysadmin -it --image=ubuntu
error: unknown profile: sysadmin
arnaldo2792 commented 5 months ago

The profiles are version-dependent. I used kubelet v1.30.X, and it allowed me to use the sysadmin profile:

~  ❯ kubectl version
Client Version: v1.30.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.9-eks-036c24b
max-rocket-internet commented 5 months ago

@arnaldo2792 can I open a PR to update the FAQ? If so, where? It would be quite useful to share this I think.

arnaldo2792 commented 5 months ago

Of course! I think this will be the best place to document this:

https://github.com/bottlerocket-os/bottlerocket-project-website

Under FAQ items

max-rocket-internet commented 5 months ago

Alright, nice: https://github.com/bottlerocket-os/bottlerocket-project-website/pull/459