K3s Cluster created with weave don't have CNI loopback driver

[de13]

Describe the bug Similar to #596 on rke. After installed weave, coredns complains about missing loopback driver in /opt/cni/bin.

E0311 21:19:12.004535     823 kuberuntime_manager.go:677] createPodSandbox for pod "coredns-7748f7f6df-8nzpj_kube-system(fe57b2c2-4437-11e9-b3ee-b827ebd31783)" failed: rpc error: code = Unknown desc = failed to setup network for sandbox "2938859438754e66cfa02804b9d857b029f1876cdc898879ad9d6a8abe0b221e": failed to find plugin "loopback" in path [/opt/cni/bin]

To Reproduce

$ sudo k3s server --disable-agent --no-deploy traefik --container-runtime-endpoint=/run/crio/crio.sock
$ sudo k3s agent --no-flannel --server https://192.168.1.6:6443 --token $(sudo cat /var/lib/rancher/k3s/server/node-token)
$ k3s kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(k3s kubectl version | base64 | tr -d '\n')"

Expected behavior

All CNI drivers for the network

[de13]

As a workaround, this is working : https://github.com/kubernetes-sigs/cri-o/blob/master/tutorial.md#cni-plugins

[westonmyers]

I had this same behavior. I found it to be due to AMD64 CNI plugin binaries vs ARM64. https://github.com/rancher/rke/issues/1553

[alexellis]

What's the fix? Does this apply in 1.0?

[mccricardo]

Any news on this?

[westonmyers]

Looking at the build files, I don't believe that this is still an issue. Are people still running into problems?

[mccricardo]

Just tried this and it's still not working.

[westonmyers]

Just to make sure, what were your commands for starting k3s server / k3s agent ? (--no-flannel is now --flannel-backend=none for example by the way.)

Once I know your starting point versus the ticket opener, I'll poke at it on my end in a bit.

[mccricardo]

Actually trying with k3d. Sorry for the confusion. k3d create --server-arg --flannel-backend=none

[non7top]

Faced the same

# curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--flannel-backend=none" sh -

In logs

Feb 12 03:24:55 k3s k3s[23604]: E0212 03:24:55.574053   23604 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "metrics-server-6d684c7b5-vv2mm_kube-system(5fab9e22-de2c-4479-857d-a6412e512f1a)" failed: rpc error: code = Unknown desc = failed to setup network for sandbox "43f447fbd5eb66179b399f34902c83777c848e9ce0c0ed541d3155f81e583f9c": failed to find plugin "loopback" in path [/opt/cni/bin]

I had to manually install files from https://github.com/containernetworking/plugins/releases/download/v0.6.0/cni-plugins-amd64-v0.6.0.tgz

[sedlund]

I had to manually install files from https://github.com/containernetworking/plugins/releases/download/v0.6.0/cni-plugins-amd64-v0.6.0.tgz

Thanks I extracted the latest tgz from above link into /opt/cni/bin and this allowed istio cni to come up.

[brandond]

@sedlund we package cni-plugins v0.8.6 - what version did you install to get this working?

[sedlund]

@brandond I installed with --flannel-backend=none and there is no /opt/cni directory after install. I used 0.9.1 and put them on /opt/cni/bin

[brandond]

Ah I see, yes. We don't put them in /opt/cni/bin, they're in /var/lib/rancher/k3s/data/current/bin alongside all our other userspace binaries. Our packaged flannel is configured to look there automatically, so if you install your own CNI you will need to grab the plugins yourself or point it at the correct path to find ours.

[sedlund]

I tried making symlinks, but noticed the firewall cni was missing so had to grab the tarball still for istio.

[brandond]

Ahh yeah that is not one we wrap, see: https://github.com/rancher/plugins/commit/e85b4849afe8ed33bb5f7d6419c702e405b0bd2d#diff-2873f79a86c0d8b3335cd7731b0ecf7dd4301eb19a82ef7a1cba7589b5252261R17-R21

[caroline-suse-rancher]

Closing this due to age