KeyError on multiple queries

[hryamzik]

I have two different queries on a single graph:

• alias(scaleToSeconds(sumSeries(stats.starman.request.count.*), 1), 'rps/all')
• alias(scaleToSeconds(sumSeries(stats.starman.request.ke_import_count.*), 1), 'extern/all)')

They both work individually but fail if requested together.

Here're the request details from grafana:

Request details

Url /api/datasources/proxy/4/render
Method  POST
Content-Type    application/x-www-form-urlencoded
Accept  application/json, text/plain, */*
Request parameters

target  alias(scaleToSeconds(sumSeries(stats.starman.request.count.*), 1), 'rps/all')
target  alias(scaleToSeconds(sumSeries(stats.starman.request.ke_import_count.*), 1), 'extern/all)')
from    -12h
until   now
format  json
maxDataPoints   946

And here's the error:

Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1817, in wsgi_app response = self.full_dispatch_request() File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1477, in full_dispatch_request rv = self.handle_user_exception(e) File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1381, in handle_user_exception reraise(exc_type, exc_value, tb) File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1475, in full_dispatch_request rv = self.dispatch_request() File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1461, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args) File "/usr/local/lib/python2.7/dist-packages/graphite_api/app.py", line 375, in render data_store = fetchData(context, paths) File "/usr/local/lib/python2.7/dist-packages/graphite_api/render/datalib.py", line 160, in fetchData time_info, series = finder.fetch_multi(nodes, startTime, endTime) File "/usr/local/lib/python2.7/dist-packages/cyanite.py", line 125, in fetch_multi time_info = data['from'], data['to'], data['step'] KeyError: 'from'

[brutasse]

@hryamzik which version of cyanite do you have? I tried the same queries on my cyanite setup (pip versions of graphite-api and cyanite, pyr/cyanite master) and couldn't reproduce the issue.

[hryamzik]

Both latest from master. I'll give it a try in vagrant a bit later.

[brutasse]

Ok. If I can have access to a setup that reproduces the issue it'd be great… I won't be available this weekend though.

[hryamzik]

It's OK, have a good weekend! I'll try to get a demo by Monday.

[hryamzik]

Here we go. Checkout this repo

Once you'll get up and running, fill in the data, I use two files, one for StatsdClient, and another one for data generation:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from python_example import StatsdClient as statsd
import socket
import sys
import re
from random import randint
import time 

def report_count(key, value, hostname):
    statsd.send({"%s.%s"%(key,hostname):"%s|c"%(value)}, (sys.argv[1], 8125))
    print "%s.%s %s"%(key,hostname,value)
    pass

def send_all(hosts, keys):
    for host in hosts:
        for key in keys:
            report_count(key, randint(0,9), host)
    pass

hostnames = [ "host1", "host2", "host3", "host4", "host5" ]
metrics = [ "starman.request.count", "starman.request.ke_import_count" ]

try:
    while True:
        send_all( hostnames, metrics )
        time.sleep(1)
except KeyboardInterrupt:
    pass

Then run it with an eth1 IP address of cyanite VM as a single argument, like this:

./fakedata.py 172.28.128.3

And create a graph with two queries:

Second graph works only when the first one is turned off:

In fact I don't see the error anymore but graphs still don't work together.

[hryamzik]

@brutasse shall I give you access to my vagrant environment?

[hryamzik]

I can still reproduce this bug with some complicated dashboards:

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python2.7/dist-packages/graphite_api/app.py", line 375, in render
    data_store = fetchData(context, paths)
  File "/usr/local/lib/python2.7/dist-packages/graphite_api/render/datalib.py", line 159, in fetchData
    time_info, series = finder.fetch_multi(nodes, startTime, endTime)
  File "/usr/local/lib/python2.7/dist-packages/cyanite.py", line 125, in fetch_multi
    time_info = data['from'], data['to'], data['step']
KeyError: 'from'

[luckyswede]

@hryamzik did you get this sorted? I face the same issue with cyanite master and latest graphite-api and graphite-cyanite from pip (1.1.2 and 0.4.6)

[hryamzik]

@luckyswede nope and due to this and other issues we've decided not to run cyanite. We are developing our own cassandra based product now.

[luckyswede]

I actually got it working by increasing the number of bytes cyanite accepts on the first http request line. It defaults to 4096 which quickly becomes too little since all metric paths are included in the get parameters... So my code change is in cyanite (http.clj where HttpServerCodec is instantiated).