Closed grebois closed 7 years ago
From where exactly do you get the error message? From kubelet running on the nodes, one of the Pods in Kubernetes, or Ansible when running the playbook?
From within the nodes or the master, running;
$ kubectl get nodes
2017-09-05 10:52:46.980000 I | proto: duplicate proto type registered: google.protobuf.Any
2017-09-05 10:52:46.980190 I | proto: duplicate proto type registered: google.protobuf.Duration
2017-09-05 10:52:46.980287 I | proto: duplicate proto type registered: google.protobuf.Timestamp
The connection to the server localhost:8080 was refused - did you specify the right host or port?
@grebois I deployed on coreos-stable-1465-6-0-v20170817 and rebooted one of the masters. It takes approximately 2 minutes after reboot for kube-apiserver to get launched and start accepting queries. There is a significant delay because the following actions take place automatically: 1 - systemd starts docker and kubelet 2 - kubelet tries to connect to apiserver if it can first 3 - the connection times out and kubelet decides to start static pods on its own 4 - kube-apiserver tries to connect to etcd 5 - etcd was restarted by docker automatically and takes some time to reassemble quorum 6 - finally kube-apiserver has a connection to etcd and will start serving requests
I observed the same behavior and timing on Ubuntu. This is not a bug, in my opinion.
If you still have issues running kubectl get nodes
after the host is up for 5 minutes, please provide complete output from the following command:
journalctl -u kubelet.service
@mattymo will try this again on a new cluster and get back to you, thanks!
@mattymo
core@int00web11 ~ $ uptime
12:44:15 up 15 min, 1 user, load average: 0.09, 0.12, 0.16
core@int00web11 ~ $ kubectl get nodes
2017-09-06 12:44:18.777759 I | proto: duplicate proto type registered: google.protobuf.Any
2017-09-06 12:44:18.777846 I | proto: duplicate proto type registered: google.protobuf.Duration
2017-09-06 12:44:18.777868 I | proto: duplicate proto type registered: google.protobuf.Timestamp
The connection to the server localhost:8080 was refused - did you specify the right host or port?
core@int00web11 ~ $ journalctl --no-pager -u kubelet.service > kubelet.log
@mattymo would you mind re-opening this issue, I'm still experiencing the same behaviour with different versions and different configurations, couldn't find the root cause yet. Thanks in advance.
DO this: (on Centos7+ and, i believe on xenial+ should worked)
$sysctl net.bridge-bridge=nf-call-iptables=1 OR $echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables $sysctl --system
I had the same issue after a clean install of kubespray, using user ubuntu on my nodes. The following fixed the issue and allowed executing kubectl commands:
Execute in master node:
sudo cp /etc/kubernetes/admin.conf $HOME/ && sudo chown $(id -u):$(id -g) $HOME/admin.conf && export KUBECONFIG=$HOME/admin.conf
I guess this should be added to the install scripts.
I'm not sure if this was fixed in the commit above, but I had to still run @jvleminc's fix for kubectl to work on Ubuntu. I'd be interested in making the fix myself, but I won't have time for the next few weeks or so.
Same here, had to run @jvleminc's workaround to see my nodes. It can't be that hard to work into the playbooks, can it?
BUG REPORT
After rebooting the master node I get:
The connection to the server localhost:8080 was refused - did you specify the right host or port?
Steps to reproduce
Restart master node.
Environment:
Bare-metal
printf "$(uname -srm)\n$(cat /etc/os-release)\n"
):Linux 4.12.7-coreos x86_64 NAME="Container Linux by CoreOS" ID=coreos VERSION=1465.6.0 VERSION_ID=1465.6.0 BUILD_ID=2017-08-16-0012 PRETTY_NAME="Container Linux by CoreOS 1465.6.0 (Ladybug)" ANSI_COLOR="38;5;75" HOME_URL="https://coreos.com/" BUG_REPORT_URL="https://issues.coreos.com" COREOS_BOARD="amd64-usr"
ansible --version
):ansible 2.3.1.0 config file = configured module search path = Default w/o overrides python version = 2.7.10 (default, Oct 23 2015, 19:19:21) [GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)]
Kubespray version (commit) (
git rev-parse --short HEAD
):fc79056
Network plugin used:
Calico