wolstena
commented
6 years ago I managed to get things working with the help of the following ticket: https://github.com/kubernetes-incubator/kube-aws/issues/1057
I deleted the default token and and the same with the service account tokens and restarted the pods.
I'm in the process of upgrading a cluster provision with kube-aws 0.9.6 to 0.9.9. My first hope in the upgrade process was from 0.9.6 to 0.9.8. This seemed to go fine. I'm having some issues getting to 0.9.9. It seems to be related to authentication issues with kube-proxy.
The environment:
17:43 $ kubectl version Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.3", GitCommit:"f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd", GitTreeState:"clean", BuildDate:"2017-11-09T07:26:41Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.4+coreos.0", GitCommit:"4292f9682595afddbb4f8b1483673449c74f9619", GitTreeState:"clean", BuildDate:"2017-11-21T17:22:25Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}After running kube-aws render stack, and kube-aws render credentials I ran kube-aws upgrade s3://...
After completing the upgrade the kube-dns and kube-dashboard pods failed. After checking a few things I noticed the following errors kube-proxy.
kubectl logs kube-proxy-x5p88 --namespace kube-system
I0216 01:20:58.562798 1 feature_gate.go:156] feature gates: map[] time="2018-02-16T01:20:58Z" level=warning msg="Running modprobe ip_vs failed with message:modprobe: ERROR: ../libkmod/libkmod.c:557 kmod_search_moddep() could not open moddep file '/lib/modules/4.14.19-coreos/modules.dep.bin', error: exit status 1" W0216 01:20:58.570929 1 server_others.go:268] Flag proxy-mode="" unknown, assuming iptables proxy I0216 01:20:58.572403 1 server_others.go:122] Using iptables Proxier. W0216 01:20:58.682921 1 server.go:580] Failed to retrieve node info: Unauthorized W0216 01:20:58.683033 1 proxier.go:468] invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP I0216 01:20:58.683125 1 server_others.go:157] Tearing down inactive rules. I0216 01:20:58.722011 1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072 I0216 01:20:58.722150 1 conntrack.go:52] Setting nf_conntrack_max to 131072 I0216 01:20:58.722277 1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400 I0216 01:20:58.722342 1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600 I0216 01:20:58.722565 1 config.go:102] Starting endpoints config controller I0216 01:20:58.722600 1 controller_utils.go:1041] Waiting for caches to sync for endpoints config controller I0216 01:20:58.722650 1 config.go:202] Starting service config controller I0216 01:20:58.722741 1 controller_utils.go:1041] Waiting for caches to sync for service config controller E0216 01:20:58.724826 1 event.go:200] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"ip-10-143-13-40.aue1.central1.cc.1513a9acbce3a724", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"ip-10-143-13-40.aue1...", UID:"ip-10-143-13-40.aue1...", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"Starting", Message:"Starting kube-proxy.", Source:v1.EventSource{Component:"kube-proxy", Host:"ip-10-143-13-40.aue1..."}, FirstTimestamp:v1.Time{Time:time.Time{sec:63654340858, nsec:722494244, loc:(*time.Location)(0x9e22280)}}, LastTimestamp:v1.Time{Time:time.Time{sec:63654340858, nsec:722494244, loc:(*time.Location)(0x9e22280)}}, Count:1, Type:"Normal"}': 'Unauthorized' (will not retry!) E0216 01:20:58.725339 1 reflector.go:205] k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:73: Failed to list *api.Endpoints: UnauthorizedOutput from kube-dns: kubectl logs kube-proxy-x5p88 --namespace kube-system
I0216 01:54:58.433213 1 dns.go:48] version: 1.14.6-3-gc36cb11 I0216 01:54:58.433907 1 server.go:69] Using configuration read from directory: /kube-dns-config with period 10s I0216 01:54:58.433943 1 server.go:112] FLAG: --alsologtostderr="false" I0216 01:54:58.433957 1 server.go:112] FLAG: --config-dir="/kube-dns-config" I0216 01:54:58.433962 1 server.go:112] FLAG: --config-map="" I0216 01:54:58.433966 1 server.go:112] FLAG: --config-map-namespace="kube-system" I0216 01:54:58.433969 1 server.go:112] FLAG: --config-period="10s" I0216 01:54:58.433976 1 server.go:112] FLAG: --dns-bind-address="0.0.0.0" I0216 01:54:58.433979 1 server.go:112] FLAG: --dns-port="10053" I0216 01:54:58.433985 1 server.go:112] FLAG: --domain="cluster.local." I0216 01:54:58.433990 1 server.go:112] FLAG: --federations="" I0216 01:54:58.433999 1 server.go:112] FLAG: --healthz-port="8081" I0216 01:54:58.434002 1 server.go:112] FLAG: --initial-sync-timeout="1m0s" I0216 01:54:58.434006 1 server.go:112] FLAG: --kube-master-url="" I0216 01:54:58.434010 1 server.go:112] FLAG: --kubecfg-file="" I0216 01:54:58.434016 1 server.go:112] FLAG: --log-backtrace-at=":0" I0216 01:54:58.434021 1 server.go:112] FLAG: --log-dir="" I0216 01:54:58.434027 1 server.go:112] FLAG: --log-flush-frequency="5s" I0216 01:54:58.434030 1 server.go:112] FLAG: --logtostderr="true" I0216 01:54:58.434033 1 server.go:112] FLAG: --nameservers="" I0216 01:54:58.434036 1 server.go:112] FLAG: --stderrthreshold="2" I0216 01:54:58.434040 1 server.go:112] FLAG: --v="2" I0216 01:54:58.434045 1 server.go:112] FLAG: --version="false" I0216 01:54:58.434052 1 server.go:112] FLAG: --vmodule="" I0216 01:54:58.434086 1 server.go:194] Starting SkyDNS server (0.0.0.0:10053) I0216 01:54:58.434303 1 server.go:213] Skydns metrics enabled (/metrics:10055) I0216 01:54:58.434312 1 dns.go:146] Starting endpointsController I0216 01:54:58.434316 1 dns.go:149] Starting serviceController I0216 01:54:58.434476 1 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0] I0216 01:54:58.434486 1 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0] I0216 01:54:58.934565 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver... I0216 01:54:59.434644 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...I'm not sure whether this is related to messed up service accounts or certs or something else.