brutasse / graphite-cyanite

A plugin for using graphite-web with the cassandra-based Cyanite storage backend.
BSD 3-Clause "New" or "Revised" License
85 stars 21 forks source link

Graphite-API returns 500 #14

Closed ghost closed 9 years ago

ghost commented 9 years ago

Hello guys,

I have to be honest: I don't know what is causing this problem, so I'm starting here and hope to get a hint or wink into the right direction. My setup look's as following:

Cassandra 2.2 Graphite-API 1.0.1 Cyanite 0.4.5 (pip install cyanite) and also pyr/cynite

wget https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein -O /usr/bin/lein && chmod 755 /usr/bin/lein && /usr/bin/lein && cd /tmp && git clone https://github.com/pyr/cyanite && cd /tmp/cyanite && /usr/bin/lein uberjar

Collectd 5.5 Grafana 2.2

vim /etc/graphite-api.yaml

cyanite:
  urls:
    - http://127.0.0.1:8080
finders:
  - cyanite.CyaniteFinder
/etc/cyanite.yaml

engine:
  rules:
    'web.*\.cpu': [ "5s:1h", "30s:1d" ]
    default: [ "5s:1h" ]
api:
  port: 8080
input:
  - type: carbon
    port: 2003
  - type: pickle
    port: 2004
index:
  type: memory
store:
  cluster: 'localhost'
  keyspace: 'metric'
logging:
  level: info
  console: true
  files:
    - "/var/log/cyanite/cyanite.log"

netstat is showing all ports that I need:

Active Internet connections (only servers)                                                                                                                                                                                                                                                                                                                         [10/1820]
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:43731         0.0.0.0:*               LISTEN      7348/java
tcp        0      0 127.0.0.1:7000          0.0.0.0:*               LISTEN      7348/java
tcp        0      0 127.0.0.1:7199          0.0.0.0:*               LISTEN      7348/java
tcp        0      0 0.0.0.0:48360           0.0.0.0:*               LISTEN      14350/beam.smp
tcp6       0      0 :::8013                 :::*                    LISTEN      16040/apache2
tcp6       0      0 :::8080                 :::*                    LISTEN      16784/java
tcp6       0      0 :::80                   :::*                    LISTEN      16040/apache2
tcp6       0      0 :::4369                 :::*                    LISTEN      14253/epmd
tcp6       0      0 127.0.0.1:9042          :::*                    LISTEN      7348/java
tcp6       0      0 127.0.0.1:2003          :::*                    LISTEN      1448/java
tcp6       0      0 127.0.0.1:2004          :::*                    LISTEN      1448/java
tcp6       0      0 :::3000                 :::*                    LISTEN      20542/grafana-serve

My tests look as following:

I'm curling the Graphite-API port on 8013, to get all metrics. Of course the Cassandra DB is empty at the beginning:

screen shot 2015-09-30 at 14 38 07

I'm just getting an [] back. After starting the collector (collectd), my curl's are producing a 500 in the apache access logs and I'm not able to query for data:

screen shot 2015-09-30 at 14 48 55

Any ideas on that?

Cheers, Szop

brutasse commented 9 years ago

If you have the latest cyanite, this is most likely this: https://github.com/pyr/cyanite/issues/130

There's a patch proposed for graphite-cyanite. Can you have a look? It could be used as a temporary solution -- there will be a proper fix in pyr/cyanite later.

ghost commented 9 years ago

Hey @brutasse,

thanks for the fast response. Actually the replies belong to me :) What patch are you talking about?

Cheers, Szop

brutasse commented 9 years ago

Ah sorry, wrong issue! Patch is here: https://github.com/pyr/cyanite/issues/119#issuecomment-143243827

ghost commented 9 years ago

By the way. After my Apache Graphite-API started to report me 500, I'm still able to query the Cyanite API port on 8080

screen shot 2015-09-30 at 15 05 51

Cheers, Szop

brutasse commented 9 years ago

Yeah, basically the return format for cyanite's /paths endpoint changed and graphite-cyanite crashes on the new return format. Ideally the cyanite api should be updated to use the old format again but it's easier to have a workaround in graphite-cyanite.

ghost commented 9 years ago

Thanks @brutasse ! I guess this issue can be closed now.

Cheers, Szop

brutasse commented 9 years ago

Cool, thanks!