search

k8snetworkplumbingwg / multus-cni

A CNI meta-plugin for multi-homed pods in Kubernetes
Apache License 2.0
2.35k stars 585 forks source link

delegate CNI doesn't see serviceaccount token of multus daemon from chroot execution #1258

Open nvetel opened 5 months ago

nvetel commented 5 months ago

Hello,

I'm using multus thick v4.0.2 and a delegate CNI that needs to request the API: the userspace CNI

I have errors because my delegate CNI searches for the file /var/run/secrets/kubernetes.io/serviceaccount/token but doesn't find it.

I tried to understand what happens exactly, and it seems to me that the issue here is related to the "chroot envrionment" used to run the delegate CNI that can't read the expected files.

The daemon has a serviceaccount mounted in /var/run/secrets/kubernetes.io/serviceaccount that can be used to interact with the API. The userspace CNI get the pod attached to retrieve annotations, and write one to record a status as multus does with k8s.v1.cni.cncf.io/network-status, but since it is executed in chroot /hostroot, it doesn't see the serviceaccount wich is not on the host, but on the multus pod.

Is this the expected behaviour ? If yes, how the delegate CNI is supposed to have access to the API to fetch pods if needed ?

dougbtv commented 4 months ago

my delegate CNI searches for the file /var/run/secrets/kubernetes.io/serviceaccount/token

Since this is executed either on the host (via thin plugin) or within a chroot (within the thick plugin) it won't find paths that are specifically inside the container.

It's curious to me that the userspace CNI plugin is looking for a kubernetes service account token? (edit: I realize it must be looking for the network-status and querying the api for it)

dougbtv commented 4 months ago

I kind of think there needs to be changes to userspace CNI in order to support thick plugin functionality -- especially if userspace CNI was reusing the kubeconfig from multus (it should likely generate its own)

nvetel commented 3 months ago

Thanks @dougbtv for the feedback on that topic.

In just wanted to make a summary of the userspace CNI's needs with the API.

The userspace CNI needs to read the pod to get annotations and pod UID (for emptydir volume only) needed to know the volume used as a shared directory for the socketfile. It will also write a userspace-data annotation as a result of its configuration in the same fashion as multus. It seems that multus has been a source of inspiration with those annotations, as well as the multus kubeconfig reuse !

Anyway, a kubeconfig generation has to be provided in the userspace CNI, or any other CNI that needs to use the API then.

bc185174 commented 2 months ago

Any update on solution to this? Hit the issue in v4.0.2 and on master.