tnqn
commented
4 years ago Thanks @varunmar for reporting it. Your understanding and analysis is correct, antrea-agent should either look for host-local in the hardcoded path if the plugin is put in a hardcoded path, or there should be a clear instruction to support a non default cni-bin-dir. We should have a fix for it soon.
For the question, it was done in this way because it requires credentials to get node information(internalIP/externalIP/PodCIDR) from K8s APIServer to do IPAM, routing, tunnel, e.g. , also needs to enforce NetworkPolicy for Pod interfaces. So it seems good to have a single place to do all these things and keep the antrea-cni binary simple (otherwise it would need to put some credentials to host fs and multiple components would program ovsdb and openflow). The host-local plugin inside the image is indeed the same as the one in containernetworking/plugins. It was just in case user don't install cni packages on their hostfs. Does the explanation make sense to you?
varunmar
Describe the bug If kubelet is started with a cni-bin-dir different from /opt/cni/bin, the antrea-agent can not find the host-local ipam plugin. This is true even if the node itself has the plugin in the cni-bin-directory.
To Reproduce On a running GKE cluster, install antrea using the https://raw.githubusercontent.com/vmware-tanzu/antrea/master/build/yamls/antrea.yml file, but replace the host-cni-bin volume hostpath with /home/kubernetes bin.
Then try to start any pod.
Expected The pod starts.
Actual behavior The pod remains in containerCreating state forever, with the following in the logs -
Warning FailedCreatePodSandBox 82s kubelet, gke-antrea-default-pool-3945f630-16tl Failed create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "cb48229d5ebe51e0d5bce60089ddd59958274cf59e7dcc9f867687e2275d31bd" network for pod "iperf-client": NetworkPlugin cni failed to set up pod "iperf-client_default" network: failed to find plugin "host-local" in path [/home/kubernetes/bin], failed to clean up sandbox container "cb48229d5ebe51e0d5bce60089ddd59958274cf59e7dcc9f867687e2275d31bd" network for pod "iperf-client": NetworkPlugin cni failed to teardown pod "iperf-client_default" network: failed to find plugin "host-local" in path [/home/kubernetes/bin]]
Versions: Please provide the following information:
kubectl version). If your Kubernetes components have different versions, please provide the version for all of them. 1.13.11uname -r). 4.15.0-1044-gkemodinfo openvswitchfor the Kubernetes Nodes. $ modinfo openvswitch filename: /lib/modules/4.15.0-1044-gke/kernel/net/openvswitch/openvswitch.ko alias: net-pf-16-proto-16-family-ovs_meter alias: net-pf-16-proto-16-family-ovs_packet alias: net-pf-16-proto-16-family-ovs_flow alias: net-pf-16-proto-16-family-ovs_vport alias: net-pf-16-proto-16-family-ovs_datapath license: GPL description: Open vSwitch switching datapath srcversion: 63B05BEAA61A95AEA35C172 depends: nf_nat,nf_nat_ipv6,nf_nat_ipv4,nf_defrag_ipv6,nsh retpoline: Y intree: Y name: openvswitch vermagic: 4.15.0-1044-gke SMP mod_unload signat: PKCS#7 signer:sig_key:
sig_hashalgo: md4
The default antrea cni configuration looks like this: { "cniVersion":"0.3.0", "name": "antrea", "type": "antrea", "ipam": { "type": "host-local" } }
When the antrea-cni delegates to the antrea-agent, it looks inside the daemonset pod filesystem for the ipam plugin. However, it does so using the CNI_BIN_DIR it gets from kubelet, which is set to /home/kubernetes/bin for all GKE nodes. The daemonset filesystem doesn't have that directory, so the path lookup fails and the pod never gets created.
As an aside, it appears the antrea-agent implements the entire CNI runtime - why was this done, instead of letting kubelet do it and just call the right plugins? It doesn't look like the host-local plugin inside the antrea-agent daemonset is any different from the normal ones in containernetworking/plugins.
Thanks!