ACS generated k8s Nodes showing up as NOT READY

[mobeus1]

Is this a request for help?: Yes

---

Is this an ISSUE or FEATURE REQUEST? (choose one): Issue

---

What version of acs-engine?: v0.21.2

---

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm) Kubernetes 1.10.6

What happened: k8s nodes are in a NOT READY state after building k8s cluster with ACS Engine.

What you expected to happen: We have built these many times without this issue

How to reproduce it (as minimally and precisely as possible): We can reproduce this by building another cluster

Anything else we need to know: Adding details below. MS Engineer has been assigned and asked for us to open an issue as well... Looks to be a CNI issue.

Issue:

kubectl get nodes

k8s-dcomqan01-39566733-0 NotReady agent 4d v1.10.6 k8s-dcomqan01-39566733-1 NotReady agent 4d v1.10.6 k8s-master-39566733-0 NotReady master 4d v1.10.6

kubectl get pods -n kube-system 4:08 PM Sathish: kube-addon-manager-k8s-master-39566733-0 1/1 Running 1 4d kube-apiserver-k8s-master-39566733-0 1/1 Running 1 4d kube-controller-manager-k8s-master-39566733-0 0/1 CrashLoopBackOff 1020 4d kube-scheduler-k8s-master-39566733-0 1/1 Running 1 4d

Description of agent k8s-dcomqan01-39566733-0 Ready False Tue, 04 Dec 2018 21:15:25 +0000 Fri, 30 Nov 2018 17:26:18 +0000 KubeletNotReady runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

Looking into kubelet logs:

W1202 03:04:32.353899 1896 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d E1202 03:04:32.354028 1896 kubelet.go:2130] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network pl W1202 03:04:37.355336 1896 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d E1202 03:04:37.355520 1896 kubelet.go:2130] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network pl lines 1-39

/etc/cni directory does not exist. = We see that the directory exists in another cluster for the customer that is working. = We also see that the calico pods are also not running in this cluster

Looked into cluster-provisioning.log and confirmed:

• installNetworkPlugin
• [[ kubenet = \a\z\u\r\e ]]
• [[ kubenet = \k\u\b\e\n\e\t ]]
• installCNI
• mkdir -p /opt/cni/bin
• retrycmd_get_tarball 60 5 /tmp/containernetworking_cni.tgz https://acs-mirror.azureedge.net/cni/cni-plugins-amd64-v0.7.1.tgz
• tar_retries=60
• wait_sleep=5
• tarball=/tmp/containernetworking_cni.tgz
• url=https://acs-mirror.azureedge.net/cni/cni-plugins-amd64-v0.7.1.tgz
• echo '60 retries' 60 retries ++ seq 1 60
• for i in '$(seq 1 $tar_retries)'
• tar -tzf /tmp/containernetworking_cni.tgz tar (child): /tmp/containernetworking_cni.tgz: Cannot open: No such file or directory tar (child): Error is not recoverable: exiting now tar: Child returned status 2 tar: Error is not recoverable: exiting now

Confirmed that https://acs-mirror.azureedge.net/cni/cni-plugins-amd64-v0.7.1.tgz is reachable on machine.

[CecileRobertMichon]

Hi @mobeus1 can you please try with the latest acs-engine version (0.26.1). This is an old Azure CNI issue that should now be fixed.

[mobeus1]

Thanks Cecile...

Will try shortly and then update this issue ...

[SidhuG]

I have tried this with version 0.26.2 of acs-engine, and see same output as above. A bug/issue has been opened on 'aks-engine', https://github.com/Azure/aks-engine/issues/112 I am trying this on 1.12.2 version of kubernetes.

[mboersma]

Closing this issue in favor of Azure/aks-engine#112.