search

kinvolk / seccompagent

agent for handling seccomp descriptors for container runtimes
Apache License 2.0
41 stars 10 forks source link

kuberesolver: Use 'oci-annotations' to read container runtime annotations #32

Closed mqasimsarfraz closed 1 year ago

mqasimsarfraz commented 1 year ago

This PR use oci-annotations package to get pod context form container runtime annotations. The changes itself are simple but vendor changes are making them blotted. Please let me know if you have any feedback on that.

Testing done

# spinning up minikube with custom image with latest stable docker v23.0.0
$ minikube start --driver=docker --container-runtime=containerd --base-image ghcr.io/mqasimsarfraz/kickbase:v0.0.37-1675280603-15763
# copy seccomp profile to prepare kubelet
$ docker exec -it minikube mkdir -p /var/lib/kubelet/seccomp/
$ docker cp docs/profiles/notify-dangerous.json minikube:/var/lib/kubelet/seccomp/
# deploy agent and test pod (need to build/load image and modify deployment for PR)
$ kubectl apply -f deploy/seccompagent.yaml
$ kubectl apply -f docs/examples/pod.yaml

You can see from the logs that pod context is being resolved via annotations. Also, containerd namespace annotation will be used as well.

Logs

before

→ kubectl logs -n seccomp-agent seccomp-agent-h5ln8 
time="2023-02-09T10:09:29Z" level=debug msg="New seccomp fd received on socket" annotations="map[io.kubernetes.cri.container-name:container1 io.kubernetes.cri.container-type:container io.kubernetes.cri.image-name:docker.io/library/busybox:latest io.kubernetes.cri.sandbox-id:0f90b1485cadc293c9b2f1409628b8416b892e12fd50c3a1c1c3479f0698f03c io.kubernetes.cri.sandbox-name:mypod io.kubernetes.cri.sandbox-namespace:default io.kubernetes.cri.sandbox-uid:81999c80-e69e-42ad-9d07-827435e4ebcd]" fd=7 id=2f998400cc9a018527d14905267e002d6100b6b307181dd1f8b9befa4d16ef40 pid=3344 pid1=3344
time="2023-02-09T10:09:29Z" level=trace msg="Found pod from pid" cgroupPathV1= cgroupPathV2=/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81999c80_e69e_42ad_9d07_827435e4ebcd.slice/cri-containerd-2f998400cc9a018527d14905267e002d6100b6b307181dd1f8b9befa4d16ef40.scope namespace=default pid=3344 pod=mypod
time="2023-02-09T10:09:29Z" level=debug msg="New container" metadata="map[DEFAULT_ACTION:kill-container MIDDLEWARE:falco]" pod="&{default mypod container1 3344 3344}"

after

→ kubectl logs -n seccomp-agent seccomp-agent-lgscj
time="2023-02-09T10:30:19Z" level=debug msg="New seccomp fd received on socket" annotations="map[io.kubernetes.cri.container-name:container1 io.kubernetes.cri.container-type:container io.kubernetes.cri.image-name:docker.io/library/busybox:latest io.kubernetes.cri.sandbox-id:4652e5ccfc132026a13d530020fe02872a5f52e4ae26b58ffbdacab44a95c9c1 io.kubernetes.cri.sandbox-name:mypod io.kubernetes.cri.sandbox-namespace:default io.kubernetes.cri.sandbox-uid:7b83445d-97ea-4404-85a0-b77878940df2]" fd=7 id=58d0417fdb0e583d421ca979151d8737fe15a0a2834fb8308f096a6691f52b51 pid=4306 pid1=4306
time="2023-02-09T10:30:19Z" level=trace msg="Pod details found from annotations" container=container1 err="<nil>" namespace=default pod=mypod
time="2023-02-09T10:30:19Z" level=debug msg="New container" metadata="map[DEFAULT_ACTION:kill-container MIDDLEWARE:falco]" pod="&{default mypod container1 4306 4306}"

crio

Works fine on crio as well:

time="2023-02-09T16:58:56Z" level=debug msg="New seccomp fd received on socket" annotations="map[io.container.manager:cri-o io.kubernetes.container.hash:68792412 io.kubernetes.container.name:container1 io.kubernetes.container.restartCount:0 io.kubernetes.container.terminationMessagePath:/dev/termination-log io.kubernetes.container.terminationMessagePolicy:File io.kubernetes.cri-o.Annotations:{\"io.kubernetes.container.hash\":\"68792412\",\"io.kubernetes.container.restartCount\":\"0\",\"io.kubernetes.container.terminationMessagePath\":\"/dev/termination-log\",\"io.kubernetes.container.terminationMessagePolicy\":\"File\",\"io.kubernetes.pod.terminationGracePeriod\":\"30\"} io.kubernetes.cri-o.ContainerID:da3ed012618185c20063671eb36cd6bce8222b5995b49d521a0e0133c4b73b0d io.kubernetes.cri-o.ContainerType:container io.kubernetes.cri-o.Created:2023-02-09T16:58:56.52144096Z io.kubernetes.cri-o.IP.0:10.244.0.12 io.kubernetes.cri-o.Image:430e2ddb3f8baa1466161e7d5287ffd730a2d737214535da4e0fae894030b278 io.kubernetes.cri-o.ImageName:quay.io/kinvolk/seccompagent:main-dfdc62e io.kubernetes.cri-o.ImageRef:430e2ddb3f8baa1466161e7d5287ffd730a2d737214535da4e0fae894030b278 io.kubernetes.cri-o.Labels:{\"io.kubernetes.container.name\":\"container1\",\"io.kubernetes.pod.name\":\"mypod\",\"io.kubernetes.pod.namespace\":\"default\",\"io.kubernetes.pod.uid\":\"27bb02ba-ee5c-47f9-9557-3f10db4400f9\"} io.kubernetes.cri-o.LogPath:/var/log/pods/default_mypod_27bb02ba-ee5c-47f9-9557-3f10db4400f9/container1/0.log io.kubernetes.cri-o.Metadata:{\"name\":\"container1\"} io.kubernetes.cri-o.MountPoint:/var/lib/containers/storage/overlay/3f73065b571bc8070b1357e1d9e4d1e06b3747f41005437f00d9923b7348a2d2/merged io.kubernetes.cri-o.Name:k8s_container1_mypod_default_27bb02ba-ee5c-47f9-9557-3f10db4400f9_0 io.kubernetes.cri-o.ResolvPath:/run/containers/storage/overlay-containers/b75102f9548f54e954506d5600cc2108c9a09203919cd1b7015eccf7b9808d47/userdata/resolv.conf io.kubernetes.cri-o.SandboxID:b75102f9548f54e954506d5600cc2108c9a09203919cd1b7015eccf7b9808d47 io.kubernetes.cri-o.SandboxName:k8s_mypod_default_27bb02ba-ee5c-47f9-9557-3f10db4400f9_0 io.kubernetes.cri-o.SeccompProfilePath:localhost//var/lib/kubelet/seccomp/notify-dangerous.json io.kubernetes.cri-o.Stdin:false io.kubernetes.cri-o.StdinOnce:false io.kubernetes.cri-o.TTY:false io.kubernetes.cri-o.Volumes:[{\"container_path\":\"/etc/hosts\",\"host_path\":\"/var/lib/kubelet/pods/27bb02ba-ee5c-47f9-9557-3f10db4400f9/etc-hosts\",\"readonly\":false,\"propagation\":0,\"selinux_relabel\":false},{\"container_path\":\"/dev/termination-log\",\"host_path\":\"/var/lib/kubelet/pods/27bb02ba-ee5c-47f9-9557-3f10db4400f9/containers/container1/7886aa8a\",\"readonly\":false,\"propagation\":0,\"selinux_relabel\":false},{\"container_path\":\"/var/run/secrets/kubernetes.io/serviceaccount\",\"host_path\":\"/var/lib/kubelet/pods/27bb02ba-ee5c-47f9-9557-3f10db4400f9/volumes/kubernetes.io~projected/kube-api-access-r4w6s\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":false}] io.kubernetes.pod.name:mypod io.kubernetes.pod.namespace:default io.kubernetes.pod.terminationGracePeriod:30 io.kubernetes.pod.uid:27bb02ba-ee5c-47f9-9557-3f10db4400f9 kubectl.kubernetes.io/last-applied-configuration:{\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"annotations\":{},\"name\":\"mypod\",\"namespace\":\"default\"},\"spec\":{\"containers\":[{\"args\":[\"-c\",\"sleep infinity\"],\"command\":[\"sh\"],\"image\":\"quay.io/kinvolk/seccompagent:main-dfdc62e\",\"imagePullPolicy\":\"Never\",\"name\":\"container1\"}],\"restartPolicy\":\"Never\",\"securityContext\":{\"seccompProfile\":{\"localhostProfile\":\"notify-dangerous.json\",\"type\":\"Localhost\"}}}}\n kubernetes.io/config.seen:2023-02-09T16:58:56.157584977Z kubernetes.io/config.source:api org.systemd.property.After:['crio.service'] org.systemd.property.CollectMode:'inactive-or-failed' org.systemd.property.DefaultDependencies:true org.systemd.property.TimeoutStopUSec:uint64 30000000]" fd=7 id=da3ed012618185c20063671eb36cd6bce8222b5995b49d521a0e0133c4b73b0d pid=7144 pid1=7144
time="2023-02-09T16:58:56Z" level=trace msg="Pod details found from annotations" container=container1 err="<nil>" namespace=default pod=mypod
time="2023-02-09T16:58:56Z" level=debug msg="New container" metadata="map[DEFAULT_ACTION:kill-container MIDDLEWARE:falco]" pod="&{default mypod container1 7144 7144}"
mqasimsarfraz commented 1 year ago

Which version of cri-o/runc did you use? Was it compiled with a sufficiently recent version of github.com/opencontainers/runtime-spec?

docker run --rm -it --entrypoint=/usr/lib/cri-o-runc/sbin/runc ghcr.io/mqasimsarfraz/kickbase:v0.0.37-1675280603-15763 --version
runc version v1.0.1
spec: 1.0.2-dev
go: go1.13.8
libseccomp: 2.5.1

You are right, it seems old. Now I remember cri-o uses custom runc in minikube (driver=docker). After switching to updated runc version I was able to test changes on cri-o as well.