exception

[creatica-soft]

Exception happened during processing of request from ('10.244.3.1', 50760) Traceback (most recent call last): File "/usr/local/lib/python3.6/socketserver.py", line 654, in process_request_thread self.finish_request(request, client_address) File "/usr/local/lib/python3.6/socketserver.py", line 364, in finish_request self.RequestHandlerClass(request, client_address, self) File "/usr/local/lib/python3.6/socketserver.py", line 724, in init self.handle() File "/usr/local/lib/python3.6/http/server.py", line 418, in handle self.handle_one_request() File "/usr/local/lib/python3.6/http/server.py", line 406, in handle_one_request method() File "/usr/local/lib/python3.6/site-packages/prometheus_client/exposition.py", line 95, in do_GET output = generate_latest(registry) File "/usr/local/lib/python3.6/site-packages/prometheus_client/exposition.py", line 69, in generate_latest for metric in registry.collect(): File "/usr/local/lib/python3.6/site-packages/prometheus_client/core.py", line 102, in collect for metric in collector.collect(): File "app.py", line 83, in collect ret = self.kube_metrics() File "app.py", line 66, in kube_metrics pod_data = session.get(self.set_namespaced_pod_url(namespace), headers=headers, params=query).json() File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 897, in json return complexjson.loads(self.text, **kwargs) File "/usr/local/lib/python3.6/json/init.py", line 354, in loads return _default_decoder.decode(s) File "/usr/local/lib/python3.6/json/decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/local/lib/python3.6/json/decoder.py", line 357, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[creatica-soft]

Lots of those exception in k8s v1.17, the pod restarts every few minutes, if not often. This is from the image v0.0.7. I tried the master one, even worse.

[creatica-soft]

Found from https://requests.readthedocs.io/en/master/api/#requests.Response.content: " json(kwargs) Returns the json-encoded content of a response, if any Parameters: kwargs – Optional arguments that json.loads takes. Raises:: ValueError – If the response body does not contain valid json." Maybe "try … except" construct is a solution to pod_data = session.get(self.set_namespaced_pod_url(namespace), headers=headers, params=query).json()

[creatica-soft]

I was able to fix that execption by wrapping all the calls to json() within try... except construct. But this is not the only exceptioon I'm getting. I'll create a separate issue for it.

[amitsingla]

@creatica-soft I am also facing the similar issue on image v0.0.7. Could you please share the try... except construct you added to get rid of this issue?.

Note: I am facing this issue only on Kubelet v1.11.7 but not on 1.12.9 . Not sure if kubelet version has any relation with this issue.

Logs: raceback (most recent call last): File "app.py", line 137, in REGISTRY.register(MetricsServerExporter()) File "/usr/local/lib/python3.6/site-packages/prometheus_client/core.py", line 54, in register names = self._get_names(collector) File "/usr/local/lib/python3.6/site-packages/prometheus_client/core.py", line 91, in _get_names for metric in desc_func(): File "app.py", line 83, in collect ret = self.kube_metrics() File "app.py", line 66, in kube_metrics pod_data = session.get(self.set_namespaced_pod_url(namespace), headers=headers, params=query).json() File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 897, in json return complexjson.loads(self.text, **kwargs) File "/usr/local/lib/python3.6/json/init.py", line 354, in loads return _default_decoder.decode(s) File "/usr/local/lib/python3.6/json/decoder.py", line 342, in decode raise JSONDecodeError("Extra data", s, end) json.decoder.JSONDecodeError: Extra data: line 1 column 5 (char 4)

[creatica-soft]

@creatica-soft I am also facing the similar issue on image v0.0.7. Could you please share the try... except construct you added to get rid of this issue?.

@amitsingla I ended up using cadvisor - a much better alternative for pod metrics. Even if you fix this issue with try..except, you'll run into broken pipe one, not worth it.

[amitsingla]

@creatica-soft Thanks for your prompt reply. We tried to setup cadvisor but unable to get the pod metrics from there. It will be great if you can share how you setup cadvisor.

[creatica-soft]

@amitsingla I can't recall that I needed to do anything special with setting cadvisor. It's just worked out of the box but have a look at my wiki notes - https://wiki.creatica.org/cgi-bin/wiki.pl/Monitoring_k8s_cluster_with_Prometheus

[amitsingla]

Thanks @creatica-soft . I will take a look.