Kube-proxy got error status on WatchServices and WatchEndpoints

mattma commented 9 years ago

core@kube-node-02 ~ $ sudo systemctl status -l kube-proxy
● kube-proxy.service - Kubernetes Proxy
   Loaded: loaded (/run/fleet/units/kube-proxy.service; linked-runtime; vendor preset: disabled)
   Active: active (running) since Thu 2015-08-20 03:53:41 UTC; 1h 22min ago
     Docs: https://github.com/GoogleCloudPlatform/kubernetes,http://kubernetes.io/v1.0/docs/admin/kube-proxy.html
  Process: 4010 ExecStartPre=/usr/bin/chmod +x /opt/bin/kube-proxy (code=exited, status=0/SUCCESS)
  Process: 3994 ExecStartPre=/usr/bin/curl -L -o /opt/bin/kube-proxy -z /opt/bin/kube-proxy https://storage.googleapis.com/kubernetes-release/release/v1.0.1/bin/linux/amd64/kube-proxy (code=exited, status=0/SUCCESS)
  Process: 3992 ExecStartPre=/usr/bin/rm /opt/bin/kube-proxy (code=exited, status=0/SUCCESS)
  Process: 3989 ExecStartPre=/usr/bin/mkdir -p /opt/bin (code=exited, status=0/SUCCESS)
 Main PID: 4013 (kube-proxy)
   Memory: 4.3M
      CPU: 9.706s
   CGroup: /system.slice/kube-proxy.service
           └─4013 /opt/bin/kube-proxy --master=http://172.17.8.100:8080 --logtostderr=true

Aug 20 03:53:13 kube-node-02 systemd[1]: Starting Kubernetes Proxy...
Aug 20 03:53:13 kube-node-02 curl[3994]: Warning: Illegal date format for -z, --timecond (and not a file name).
Aug 20 03:53:13 kube-node-02 curl[3994]: Warning: Disabling time condition. See curl_getdate(3) for valid date syntax.
Aug 20 03:53:13 kube-node-02 curl[3994]: % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Aug 20 03:53:13 kube-node-02 curl[3994]: Dload  Upload   Total   Spent    Left  Speed
Aug 20 03:53:38 kube-node-02 curl[3994]: [1.9K blob data]
Aug 20 03:53:41 kube-node-02 curl[3994]: [320B blob data]
Aug 20 03:53:41 kube-node-02 systemd[1]: Started Kubernetes Proxy.
Aug 20 05:02:04 kube-node-02 kube-proxy[4013]: W0820 05:02:04.397432    4013 api.go:153] Got error status on WatchServices channel: &{TypeMeta:{Kind: APIVersion:} ListMeta:{SelfLink: ResourceVersion:} Status:Failure Message:401: The event in requested index is outdated and cleared (the requested history has been cleared [58876/51366]) [59875] Reason: Details:<nil> Code:0}
Aug 20 05:06:22 kube-node-02 kube-proxy[4013]: W0820 05:06:22.938093    4013 api.go:224] Got error status on WatchEndpoints channel: &{TypeMeta:{Kind: APIVersion:} ListMeta:{SelfLink: ResourceVersion:} Status:Failure Message:401: The event in requested index is outdated and cleared (the requested history has been cleared [59748/59285]) [60747] Reason: Details:<nil> Code:0}

core@kube-node-02 ~ $ sudo journalctl -u kube-proxy
-- Logs begin at Tue 2015-08-18 17:51:34 UTC, end at Thu 2015-08-20 05:23:19 UTC. --
Aug 18 17:57:16 kube-node-02 systemd[1]: Starting Kubernetes Proxy...
Aug 18 17:57:16 kube-node-02 rm[1358]: /usr/bin/rm: cannot remove '/opt/bin/kube-proxy': No such file or directory
Aug 18 17:57:16 kube-node-02 curl[1361]: Warning: Illegal date format for -z, --timecond (and not a file name).
Aug 18 17:57:16 kube-node-02 curl[1361]: Warning: Disabling time condition. See curl_getdate(3) for valid date syntax.
Aug 18 17:57:16 kube-node-02 curl[1361]: % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Aug 18 17:57:16 kube-node-02 curl[1361]: Dload  Upload   Total   Spent    Left  Speed
Aug 18 17:57:17 kube-node-02 curl[1361]: [234B blob data]
Aug 18 17:57:17 kube-node-02 systemd[1]: Started Kubernetes Proxy.
Aug 18 18:51:38 kube-node-02 kube-proxy[1364]: W0818 18:51:38.734295    1364 api.go:153] Got error status on WatchServices channel: &{TypeMeta:{Kind: APIVersion:} ListMeta:{SelfLink: ResourceVersion:} Status:Failure Message:401: The event in requested in
Aug 18 19:05:36 kube-node-02 kube-proxy[1364]: W0818 19:05:36.534740    1364 api.go:224] Got error status on WatchEndpoints channel: &{TypeMeta:{Kind: APIVersion:} ListMeta:{SelfLink: ResourceVersion:} Status:Failure Message:401: The event in requested i
Aug 18 20:01:28 kube-node-02 kube-proxy[1364]: W0818 20:01:28.246425    1364 api.go:153] Got error status on WatchServices channel: &{TypeMeta:{Kind: APIVersion:} ListMeta:{SelfLink: ResourceVersion:} Status:Failure Message:401: The event in requested in
Aug 18 20:04:27 kube-node-02 kube-proxy[1364]: W0818 20:04:27.595492    1364 api.go:224] Got error status on WatchEndpoints channel: &{TypeMeta:{Kind: APIVersion:} ListMeta:{SelfLink: ResourceVersion:} Status:Failure Message:401: The event in requested i
Aug 18 22:35:56 kube-node-02 kube-proxy[1364]: W0818 22:35:56.749712    1364 api.go:153] Got error status on WatchServices channel: &{TypeMeta:{Kind: APIVersion:} ListMeta:{SelfLink: ResourceVersion:} Status:Failure Message:401: The event in requested in
Aug 18 23:03:50 kube-node-02 kube-proxy[1364]: W0818 23:03:50.948246    1364 api.go:224] Got error status on WatchEndpoints channel: &{TypeMeta:{Kind: APIVersion:} ListMeta:{SelfLink: ResourceVersion:} Status:Failure Message:401: The event in requested i
Aug 19 05:43:46 kube-node-02 kube-proxy[1364]: W0819 05:43:46.742629    1364 api.go:224] Got error status on WatchEndpoints channel: &{TypeMeta:{Kind: APIVersion:} ListMeta:{SelfLink: ResourceVersion:} Status:Failure Message:401: The event in requested i
Aug 19 05:46:01 kube-node-02 kube-proxy[1364]: W0819 05:46:01.276691    1364 api.go:153] Got error status on WatchServices channel: &{TypeMeta:{Kind: APIVersion:} ListMeta:{SelfLink: ResourceVersion:} Status:Failure Message:401: The event in requested in
Aug 19 06:30:58 kube-node-02 kube-proxy[1364]: W0819 06:30:58.203745    1364 api.go:153] Got error status on WatchServices channel: &{TypeMeta:{Kind: APIVersion:} ListMeta:{SelfLink: ResourceVersion:} Status:Failure Message:401: The event in requested in
Aug 19 06:40:10 kube-node-02 kube-proxy[1364]: W0819 06:40:10.843763    1364 api.go:224] Got error status on WatchEndpoints channel: &{TypeMeta:{Kind: APIVersion:} ListMeta:{SelfLink: ResourceVersion:} Status:Failure Message:401: The event in requested i
Aug 20 03:46:46 kube-node-02 kube-proxy[1364]: E0820 03:46:46.830407    1364 proxysocket.go:99] Dial failed: dial tcp 10.244.45.2:27017: connection refused
Aug 20 03:46:46 kube-node-02 kube-proxy[1364]: E0820 03:46:46.831744    1364 proxysocket.go:99] Dial failed: dial tcp 10.244.45.2:27017: connection refused
Aug 20 03:46:46 kube-node-02 kube-proxy[1364]: E0820 03:46:46.831787    1364 proxysocket.go:99] Dial failed: dial tcp 10.244.45.2:27017: connection refused
Aug 20 03:46:46 kube-node-02 kube-proxy[1364]: E0820 03:46:46.831823    1364 proxysocket.go:99] Dial failed: dial tcp 10.244.45.2:27017: connection refused
Aug 20 03:46:46 kube-node-02 kube-proxy[1364]: E0820 03:46:46.833005    1364 proxysocket.go:133] Failed to connect to balancer: failed to connect to an endpoint.
Aug 20 03:46:46 kube-node-02 kube-proxy[1364]: E0820 03:46:46.834083    1364 proxysocket.go:99] Dial failed: dial tcp 10.244.45.2:27017: connection refused
Aug 20 03:46:46 kube-node-02 kube-proxy[1364]: E0820 03:46:46.834127    1364 proxysocket.go:99] Dial failed: dial tcp 10.244.45.2:27017: connection refused
Aug 20 03:46:46 kube-node-02 kube-proxy[1364]: E0820 03:46:46.834162    1364 proxysocket.go:99] Dial failed: dial tcp 10.244.45.2:27017: connection refused
Aug 20 03:46:46 kube-node-02 kube-proxy[1364]: E0820 03:46:46.834194    1364 proxysocket.go:99] Dial failed: dial tcp 10.244.45.2:27017: connection refused
Aug 20 03:46:46 kube-node-02 kube-proxy[1364]: E0820 03:46:46.834201    1364 proxysocket.go:133] Failed to connect to balancer: failed to connect to an endpoint.
....

Refer to the issue 9713 and issue 9310

v1k0d3n commented 9 years ago

experiencing the same issue. i saw where someone requested this issue be reopened if it happens again. did you ever figure out how to resolve it, @mattma?

ntquyen commented 8 years ago

I got the same error with kube-proxy in kubernetes v1.0.1

W1205 09:06:20.572062       1 api.go:153] Got error status on WatchServices channel: &{TypeMeta:{Kind: APIVersion:} ListMeta:{SelfLink: ResourceVersion:} Status:Failure Message:401: The event in requested index is outdated and cleared (the requested history has been cleared [12062594/12052864]) [12063593] Reason: Details:<nil> Code:0}

mattma / coreos-distro

Kube-proxy got error status on WatchServices and WatchEndpoints #10