Add env var CDEBUG_ROOTFS pointing to the sidecar root filesystem

mauriciopoppe commented 1 year ago

Motivation

I have a kind cluster with at least 1 worker and I'd like to use cdebug to instrument a sidecar with enough tooling to run the kubelet in debug mode.

The sidecar is built from a custom image with the following things:

A systemd service for kubelet-debug, this is similar to the kubelet systemd service but instrumented to run the kubelet through delve
An entrypoint script that's run through cdebug exec, when the entrypoint script is executed:
- it copies all the tooling to the target image, in this case dlv and the new systemd service
- stops the kubelet service and starts the kubelet-debug service

At this point a target container (e.g. kind-worker) is running the kubelet through dlv, then editors can connect to it to start a debugging session.

Steps:

Create an image for the sidecar with tooling to provide debugging tools

FROM golang:bullseye AS go
RUN apt update && apt install git -y
RUN go install github.com/go-delve/delve/cmd/dlv@v1.22.0
RUN git clone https://github.com/garabik/grc.git /go/src/github.com/garabik/grc

FROM debian:bullseye
RUN apt update && apt install --only-upgrade bash -y
RUN mkdir -p /app/bin
COPY --from=go /go/bin/dlv /app/bin/dlv
COPY --from=go /go/src/github.com/garabik/grc /app/grc

WORKDIR /app
ADD . .

# I read https://github.com/garabik/grc/blob/master/install.sh
# to find out that $1 is the prefix where the binaries (grcat, grc)
# will be installed into.
RUN (cd ./grc && ./install.sh /app/)

ENTRYPOINT ["sleep", "infinity"]

In the Dockerfile workspace there are additional files related to the kubeadm setup of the kubelet

10-kubeadm.conf

Click me

``` # Note: This dropin only works with kubeadm and kubelet v1.11+ [Service] Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf" Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml" # This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env # This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use # the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file. EnvironmentFile=-/etc/default/kubelet ExecStart= # (override) I got the delve args by looking at how skaffold sets up debugging. ExecStart=dlv exec --headless --accept-multiclient --listen=:56268 --api-version=2 /usr/bin/kubelet-debug -- $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS --v=2 ```

kubelet-debug-entrypoint.sh

Click me

```bash #!/bin/bash set -euo pipefail install() { # setup systemd kubelet-debug.service unit systemd=/etc/systemd/system mkdir -p "${systemd}/kubelet-debug.service.d" cp $CDEBUG_ROOTFS/app/kubelet-debug.service "${systemd}" cp $CDEBUG_ROOTFS/app/10-kubeadm.conf "${systemd}/kubelet-debug.service.d/10-kubeadm.conf" cp $CDEBUG_ROOTFS/app/conf.kubernetes "${systemd}/kubelet-debug.service.d/conf.kubernetes" # copy dlv if not already there cp $CDEBUG_ROOTFS/app/bin/dlv /usr/bin/dlv # copy tooling (grcat) cp $CDEBUG_ROOTFS/app/bin/grcat /usr/bin/grcat cp $CDEBUG_ROOTFS/app/bin/grc /usr/bin/grc if ! command -v python3 &> /dev/null; then apt update && apt install -y python3 fi # it's assumed that the kubelet-debug binary will be replaced # later with a version of the kubelet compiled in debug mode. # # start kubelet-debug unit and disable kubelet unit systemctl daemon-reload systemctl disable kubelet && systemctl stop kubelet systemctl enable kubelet-debug && systemctl start kubelet-debug # Success message (tput setaf 2; \ echo "kind-worker patched with new kubelet!"; \ echo "next step: copy the kubelet binary compiled with debug symbols to /usr/bin/kubelet-debug"; \ echo ""; \ echo "Keep this terminal alive while you're on your debugging session."; \ tput sgr0) } restore() { # restore kubelet systemd unit systemctl disable kubelet-debug && systemctl stop kubelet-debug systemctl enable kubelet && systemctl start kubelet } install trap restore exit bash ```

kubelet-debug.service

Click me

``` [Unit] Description=kubelet: The Kubernetes Node Agent Documentation=http://kubernetes.io/docs/ # NOTE: kind deviates from upstream here to avoid crashlooping # This does *not* support altering the kubelet config path though. # We intend to upstream this change but first need to solve the upstream # Packaging problem (all kubernetes versions use the same files out of tree). ConditionPathExists=/var/lib/kubelet/config.yaml [Service] ExecStart=/usr/bin/kubelet Restart=always StartLimitInterval=0 # NOTE: kind deviates from upstream here with a lower RestartSecuse RestartSec=1s # And by adding the [Service] lines below CPUAccounting=true MemoryAccounting=true Slice=kubelet.slice KillMode=process [Install] WantedBy=multi-user.target ```

Build the image: docker build -t kubelet-debug:latest .
Run cdebug targeting the kind-worker container cdebug exec --image kubelet-debug:latest -it docker://kind-worker '$CDEBUG_ROOTFS/app/kubelet-debug-entrypoint.sh'
Exec into the kind-worker container and check the status of the systemd services

○ kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/etc/systemd/system/kubelet.service; disabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/kubelet.service.d
             └─10-kubeadm.conf
     Active: inactive (dead)
       Docs: http://kubernetes.io/docs/

root@kind-worker:/# systemctl status kubelet-debug
● kubelet-debug.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/etc/systemd/system/kubelet-debug.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/kubelet-debug.service.d
             └─10-kubeadmin.conf
     Active: active (running) since Thu 2023-01-26 05:02:50 UTC; 13s ago

Check that the dlv server started

root@kind-worker:/# journalctl -u kubelet-debug
...
Jan 26 05:02:50 kind-worker dlv[632591]: API server listening at: [::]:56268
...

At this point we can start a debugging session
- At this point is where I'd sync the kubelet compiled in debug mode (I wrote the instructions on how to do that here: https://github.com/mauriciopoppe/kubernetes-playground/blob/master/docs/kubelet.md#normal-workflow) and I'd sync it to the directory created by the sidecar and restart the kubelet-debug service
At this point I can connect my editor to 0.0.0.0:56268 which would be forwarded to the containers delve's listening port.

breakpoints in nvim

To end the session hit ctrl-c on the terminal where the sidecar is running as part of the teardown most of the instrumentation code will be removed and it'll leave the original kubelet running as it was before

Implementation details

This is possible by creating a special envvar CDEBUG_ROOTFS that points to the location of the sidecar root filesystem, the sidecar image can use this var to install tooling on the target container as seen above in the cdebug exec execution

Tests

go install ./

make e2e-test
=== RUN   TestExecDockerRootFS
--- PASS: TestExecDockerRootFS (0.77s)

mauriciopoppe commented 1 year ago

Friendly ping, what do you think about this proposal?

iximiuz commented 1 year ago

Sorry, was (and still am) really short on time for the past few weeks. From first sight, having this flexibility makes perfect sense. But I'll have to give it another look with a clearer head. Hopefully, this weekend. Thanks for your contribution!

mauriciopoppe commented 1 year ago

Pinging back, what do you think about this proposal?

mauriciopoppe commented 5 months ago

hi @iximiuz, I saw some activity in https://github.com/iximiuz/cdebug/pull/11 so if you have time please look at this proposal, this PR is ready for review.

iximiuz commented 4 months ago

@mauriciopoppe I think I understand the need for it now, and I'm willing to merge it. Would you mind renaming the env var to something like CDEBUG_ROOTFS? The word "workspace" is a bit ambiguous.

mauriciopoppe commented 4 months ago

@iximiuz Sure, I've changed the env var to be CDEBUG_ROOTFS I also added some instructions about how I tested it in the description.

iximiuz / cdebug