grobian / carbon-c-relay

Enhanced C implementation of Carbon relay, aggregator and rewriter
Apache License 2.0
380 stars 107 forks source link

No statistics being generated #414

Closed spoofedpacket closed 4 years ago

spoofedpacket commented 4 years ago

Running a carbon-c-relay instance built from master, I don't see any of the statistics metrics for the relay showing up in graphite. The instance is receiving and sending all other metrics fine.

Here's the command line for carbon-c-relay:

/usr/bin/carbon-c-relay -f /etc/carbon-c-relay.conf -p 3333 -m -w 8 -b 2500 -q 10000000 -l /var/log/carbon-c-relay/carbon-c-relay.log

And the config:

cluster graphite
  any_of
    a.a.a.a:2003
    b.b.b.b:2003
    c.c.c.c:2003
    d.d.d.d:2003
    e.e.e.e:2003
    f.f.f.f:2003
    g.g.g.g:2003
    h.h.h.h:2003
    i.i.i.i:2003
    j.j.j.j:2003
  ;

cluster stats_cluster 
  forward
    stats-cluster:2003
  ;

statistics
  reset counters after interval
  send to stats_cluster
  ;

# filter out malformed metrics
match *
  validate ^[0-9.e+-]+\ [0-9.e+-]+$ else drop
  send to graphite
  ;

Stats metrics should show up on the stats_cluster, but there's no sign of them. I also did a tcpdump on the host and there was no indication of traffic to the stats-cluster IP. Metrics are being sent to the graphite cluster from the relay, so it appears the stats are not being generated or carbon-c-relay isn't attempting to send them.

I was using a similar config with carbon-c-relay 2.6, but without the statistics block (sending the stats to the graphite cluster) and this worked as expected.

Hopefully I'm missing something basic here :)

grobian commented 4 years ago

I see nothing wrong, so I'll have to look into why this doesn't work

grobian commented 4 years ago

out of curiosity, if you change the statistics block to

send statistics to stats_cluster;

does that make the statistic visible? (this is not the same as the counters aren't reset)

grobian commented 4 years ago

Another question, is you create testconf like this:

cluster graphite
    forward
        127.0.0.1:3002
    ;

cluster stats
    file /dev/stdout
    ;

statistics
    submit every 10 seconds
    reset counters after interval
    send to stats
    ;

match *
    validate ^[0-9.e+-]+\ [0-9.e+-]+$ else drop
    send to graphite
    ;

and run the relay on that conf: relay -f testconf do you get the statistics metrics blurbed on your screen after 10 seconds?

This works for me, I get a batch of carbon.relays..* spit out.

grobian commented 4 years ago

wait... /usr/bin/carbon-c-relay -f /etc/carbon-c-relay.conf -p 3333 -m -w 8 -b 2500 -q 10000000 -l /var/log/carbon-c-relay/carbon-c-relay.log -m needs an argument, not sure how it parses this, other tests show that it barfs about this, I don't get stats in this mode

spoofedpacket commented 4 years ago

Yep, turned out it was the -m with no argument. I removed it and the stats metrics began to flow. Thanks for the quick fix!