kubeadm: kube-dns gives rpc-error failed to start container

spnzig commented 7 years ago

Is this a BUG REPORT or FEATURE REQUEST?

Choose one: BUG REPORT

Versions

kubeadm version (use kubeadm version): v1.7.2

Environment:

Kubernetes version (use kubectl version): v1.7.2
Cloud provider or hardware configuration: Hardware configuration with 2 Virtualbox VMs
OS (e.g. from /etc/os-release): Ubuntu 16.04LTS
Kernel (e.g. uname -a): Linux 4.4.0-31-generic
Others:

What happened?

I installed kubeadm on a vm running ubuntu 16.04. On the master node i wanted to initialise kubeadm on a particular interface. So i ran kubeadm init with the parameter --apiserver-advertise-address=10.83.x.x When i do a kubectl get pods -n=kube-system i get kube-dns 0/3 rpc-error: code=2 desc=failed to start container "xxxx": error response from daemon: {"message": "cannot join network of a non running container: xxxxx"} I am also facing issues with the dashboard. The state remains in pending state from 6hours

What you expected to happen?

Proper functioning of DNS and dashboard

How to reproduce it (as minimally and precisely as possible)?

Install kubeadm v1.7.2 as per the official guide and run the command

yahonda commented 7 years ago

I recall "kube-dns" failure with "cannot join network of a non running container" when I was installing kubeadm on raspberry pi 3 using Hypriot OS Version 1.5.0 and installed flannel.

kubeadm init with --pod-network-cidr 10.244.0.0/16

$ kubeadm init --pod-network-cidr 10.244.0.0/16 --apiserver-advertise-address <IP_Adddress of WIFI network interface>

Removed KUBELET_NETWORK_ARGS entries from /etc/systemd/system/kubelet.service.d/10-kubeadm.conf and restart OS(kubernetes nodes) as suggested https://github.com/hypriot/blog/issues/56
Before

[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local"
Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"
Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0"
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_EXTRA_ARGS

After

[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local"
Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"
Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0"
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_EXTRA_ARGS

This is not Ubuntu on VirtualBox examples then it may not be related with yours but hope this helped some.

Thanks.

jamiehannaford commented 7 years ago

@spnzig Did you install a CNI plugin?

luxas commented 7 years ago

I think the problem here is that you didn't install Weave Net, Flannel, or some other networking provider as @jamiehannaford highlighted. Please reopen this issue if you have done all those steps and can reproduce this behavior with kubeadm v1.8. Thank you!

kubernetes / kubeadm