Closed wolfedale closed 5 years ago
Could you check if metrics-server is actually running in kube-system
namespace?
Yes, looks like it's running
metrics-server-57b998f5fc-mpg9w 1/1 Running 0 13m
Ok, this is really strange. I run my pipeline second time and now it's working. But I did it 3-4 times before and it was failing. Maybe indeed the metrics-server was starting or for some reason it was not ready before.
It happened again, in fact I can reproduce it every time when I'm creating new cluster. I also tracked the issue, which is like you suggested not running metrics-server. Looks like it my metrics-server is starting a while and when it's ready and I will re-run pharos then it's working. So I guess the solution might be to check if metrics-server is running and wait around 1 minute for it.
I have now seen this too, created #628 to brute-force it through. I think it might be related to the speed of network deployment.. meaning: if weave/calico deployment goes up slowly then metrics-server most likely will throw errors for a while. @jnummelin WDYT?
Not sure if it's network related, or just a bug/feature on kube api as we've seen with CRDs too. But no matter the cause, I don't currently see any other way than retrying it.
1.3.3 includes couple fixes for this, @wolfedale could you test it & report the results?
I'm running into the same issue during initial cluster creation. I can reproduce with v1.3.2
and v1.3.3
but not with earlier versions. Now bisecting between v1.3.1
and v1.3.2
.
@kdomanski v1.3.3
should retry (with backoff) ... it didn't help here?
Right, bisect points to 93b32a63b5352662992c3c22643c416bfa521f38
v1.3.3
retries, but each retry yields the same result. Probably the instance I'm running on is so slow that the retry is not enough. Also, the logs of the apiserver mention API rate limiting.
Any news on this? Unable to create cluster with >1.3.1 like kdomanski, it hangs on the configuring metrics server step. Running CentOS-7-x86_64-Minimal-1804
@voneiden @kdomanski do you have any worker nodes in cluster.yml
?
Yes, one worker. With 1.3.1 it goes up without issues.
@voneiden could you check if metrics-server deployment is actually running in the cluster?
Everything seems to start
$ kubectl -n kube-system get pods
NAME READY STATUS RESTARTS AGE
coredns-69c75956f7-qwn5n 1/1 Running 0 2m
coredns-69c75956f7-zv5r8 1/1 Running 0 2m
etcd-kube-master 1/1 Running 8 1m
kube-apiserver-kube-master 1/1 Running 0 1m
kube-controller-manager-kube-master 1/1 Running 0 1m
kube-proxy-bhl86 1/1 Running 0 2m
kube-proxy-g547j 1/1 Running 0 2m
kube-scheduler-kube-master 1/1 Running 0 1m
metrics-server-57b998f5fc-94wvk 1/1 Running 0 2m
pharos-proxy-kube-worker 1/1 Running 0 1m
weave-net-58d2s 2/2 Running 0 2m
weave-net-c22lz 2/2 Running 1 2m
$ kubectl -n kube-system get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP 6m
metrics-server ClusterIP 10.104.224.225 <none> 443/TCP 5m
The metrics-server service ClusterIP responds to curl from the worker node. From master node I get 'no route to host'.
In comparison to 1.3.1, by using curl I get "connection refused" on the master node and 'no route to host' from worker node. Not sure if relevant.
@voneiden check logs from the weave-net pods, this starts to sound like network issue.
Definitely relevant, seems to boil down at least in my case to not having set trusted_subnets - weave couldn't operate in my cluster properly without it. With that set, 1.3.3 starts fine (metrics step takes a bit longer than in 1.3.1, as was pointed out earlier).
If enabling trusted subnets fixes it, it really sounds like IPSec ESP is blocked between the nodes
@kdomanski could you check if trusted-subnets helps?
@jakolehm I can no longer reproduce the issue. The metrics server is now coming up before the retry times out.
When upgraded to the latest version of pharos-cluster I'm getting this error from my pipeline:
I'm not sure where this metrics came from, I have nothing special in my ingress.