search

raintank / graphite-api

Graphite-web, without the interface. Just the rendering HTTP API.
http://graphite-api.readthedocs.org
Apache License 2.0
1 stars 1 forks source link

all "ping performance range and loss" panels are erroring in prod #13

Closed Dieterbe closed 8 years ago

Dieterbe commented 8 years ago 
Traceback (most recent call last):
  File "/usr/share/python/graphite/local/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/share/python/graphite/local/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/share/python/graphite/local/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/share/python/graphite/local/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/share/python/graphite/local/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/share/python/graphite/local/lib/python2.7/site-packages/graphite_api/app.py", line 384, in render
    data_store = fetchData(context, paths)
  File "/usr/share/python/graphite/local/lib/python2.7/site-packages/graphite_api/render/datalib.py", line 160, in fetchData
    time_info, series = finder.fetch_multi(nodes, startTime, endTime)
  File "/usr/share/python/graphite/local/lib/python2.7/site-packages/graphite_raintank.py", line 186, in fetch_multi
    data = self.fetch_from_tank(nodes, start_time, end_time)
  File "/usr/share/python/graphite/local/lib/python2.7/site-packages/graphite_raintank.py", line 220, in fetch_from_tank
    for result in resp.json():
  File "/usr/share/python/graphite/local/lib/python2.7/site-packages/requests/models.py", line 808, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Dieterbe commented 8 years ago

this is because this graph requests a lot of different series (4x the usual), which combined with the high value of max datapoints causes it to hit the limit in MT. i've upped the MT limit, see https://github.com/raintank/raintank-metric/commit/6ff6632afe20029b55bd654971af1a90469fd12d, which should fix this. will verify in qa

as per https://github.com/raintank/graphite-raintank/issues/20 it would help if graphite-raintank passed errors back to grafana in a nicer way