search

contiv / netplugin

Container networking for various use cases
Apache License 2.0
512 stars 177 forks source link

Added support for non-docker container runtime, e.g. containerd or cri-o #1167

Closed zhouzijiang closed 5 years ago

zhouzijiang commented 5 years ago

Description of the changes

Type of fix: Bug Fix

Please describe:

when kubelet creates a pod, it will call RunPodSandbox function:

  1. when the container runtime is docker in kubelet, it has the following steps: 1.1 create the sandbox container(it will be in the new network namespace) 1.2 start the sandbox container 1.3 kubelet forks and execs binary contivk8s and pass CNI_NETNS like '/proc/37544/ns/net' 1.4 contivk8s pass '/proc/37544/ns/net' to netplugin 1.5 netplugin get the new network namespace from '/proc/37544/ns/net' 1.6 netplugin execs command like 'nsenter -t 37544 -n ip address add 192.168.1.1/24 dev eth0'

  2. when the container runtime is containerd or other in kubelet, it has the following steps: 2.1 containerd saves the current network namespace 2.2 containerd calls unshare to move it to a new network namespace 2.3 containerd creates file like '/var/run/netns/cni-eb58672e-a27e-ac5d-73dd-7ebb56282818' and bind mounts it to the new network namespace 2.4 containerd restores itself to the original network namespace 2.5 now there are no processes in the new network namespace, and we can only access the new network namespace by file '/var/run/netns/cni-eb58672e-a27e-ac5d-73dd-7ebb56282818' 2.6 containerd forks and execs binary contivk8s and pass CNI_NETNS like '/var/run/netns/cni-eb58672e-a27e-ac5d-73dd-7ebb56282818' 2.7 contivk8s pass '/var/run/netns/cni-eb58672e-a27e-ac5d-73dd-7ebb56282818' to netplugin 2.8 netplugin can not parse '/var/run/netns/cni-eb58672e-a27e-ac5d-73dd-7ebb56282818' and return error

So, our solution is changing step 2.7:

  1. contivk8s gets CNI_NETNS like '/var/run/netns/cni-eb58672e-a27e-ac5d-73dd-7ebb56282818'
  2. contivk8s gets the new network namespace from '/var/run/netns/cni-eb58672e-a27e-ac5d-73dd-7ebb56282818'
  3. contivk8s makes itself to the new network namespace
  4. contivk8s creates the new argument by code 'fmt.Sprintf("/proc/%d/ns/net", os.Getpid())'
  5. contivk8s passes '/proc/12345/ns/net' to netplugin

TODO

zhouzijiang commented 5 years ago

@tiewei @vhosakot please review it, thanks again

zhouzijiang commented 5 years ago

Thanks for submitting reviews, the logic here sound, do you mind adding some test cases ?

ok, I will add some test cases. In fact, we have run contiv with this feature in our k8s cluster for about a year

zhouzijiang commented 5 years ago

@tiewei I have added some test cases

cusri-admin commented 4 years ago

very good