logstash-plugins / logstash-output-graphite

Apache License 2.0
9 stars 15 forks source link

Graphite output sending metrics every 1 to 5 seconds #19

Open trenb opened 8 years ago

trenb commented 8 years ago

Hi! I moved this ticket from https://github.com/elastic/logstash/issues/4099 at the request of @purbon

Hello! We're seeing an issue where the graphite output for logstash 1.5.x (tried both 1.5.1 and 1.5.4) are sending metrics far too often. Our graphite flush interval is 10s, but it looks like logstash is sending data every 1 to 5 seconds. In less than 60 seconds logstash sends over 30 metrics. I haven't found a way to set a flush interval for the graphite output to possibly curb this output. I've cleaned up the tcpdump data to only show the timestamp and the metric value. I've also attached the relevant sections of the configuration file.

Please let me know what other information you need to help resolve this issue.

Time Stamp Metric value
18:36:06.218506 2014.0694022039097
18:36:09.109487 0.0003298800007348804
18:36:09.126179 0.01725653926907217
18:36:11.218657 1952.9458036527249
18:36:14.109839 0.00027923739202897867
18:36:14.126638 0.01725653926907217
18:36:16.218971 0.0
18:36:19.108619 0.00027923739202897867
18:36:19.126874 0.014607345126126827
18:36:21.219442 2057.4
18:36:24.109335 0.00025691080289192054
18:36:24.127625 0.014607345126126827
18:36:26.218453 2029.6873941105234
18:36:29.109612 0.0002363693492586465
18:36:29.127553 0.012364850698436342
18:36:31.218892 2034.6696349903889
18:36:34.109475 0.0002363693492586465
18:36:34.126486 0.012364850698436342
18:36:36.218827 2005.5122608578213
18:36:39.109726 0.0002000823344717311
18:36:39.126732 0.01046662014722731
18:36:41.219014 1971.8579746512391
18:36:44.109592 0.0002000823344717311
18:36:44.126960 0.01046662014722731
18:36:46.219230 1926.1987000074062
18:36:49.109215 0.00018408463429671232
18:36:49.126185 0.009629755406503232
18:36:51.218614 1916.620124821668
18:36:54.109384 0.00016936603960377174
18:36:54.126794 0.009629755406503232
18:36:56.218765 1910.6858112952668
18:36:59.109545 0.00016936603960377174
18:36:59.127051 0.008859802675999827
# filter_10_metrics_add_metrics
filter {
 metrics {
  add_tag => ['metric']
  meter => ['events']
  clear_interval => 60
 }
}

output {
 if 'metric' in [tags] {
    graphite {
        host => 'graphite.xxx.com'
        metrics => ["dw13_xxx_com.consumed.event_rate", "%{events.rate_1m}"]
    }
 }
}
wiibaa commented 8 years ago

@jordansissel as this ticket is strongly related to flushing in the metrics filter and judging by your comment in https://github.com/elastic/logstash/issues/1231#issuecomment-193951621 I do not see a strong reason to keep this one. yes / no ?