search

influxdata / telegraf

Agent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.
https://influxdata.com/telegraf
MIT License
14.9k stars 5.6k forks source link

Aggregator documentation: historical data not supported #1989

Closed toni-moreno closed 8 years ago

toni-moreno commented 8 years ago

Bug report

Telegraf is not detecting aggregators plugins with this error.

c:\Program Files\Telegraf> C:\"Program Files"\Telegraf\telegraf.exe --config C:\"Program Files"\Telegraf\telegraf.conf --test
2016/11/03 15:17:35 Using config file: C:\Program Files\Telegraf\telegraf.conf
Error parsing C:\Program Files\Telegraf\telegraf.conf, Undefined but requested input: aggregators

Relevant telegraf.conf:

[global_tags]

# Configuration for telegraf agent
[agent]
  interval = "10s"
  round_interval = true
  metric_buffer_limit = 1000
  flush_buffer_when_full = true
  collection_jitter = "0s"
  flush_interval = "60s"
  flush_jitter = "0s"

  debug = false
  quiet = false
  #hostname = ""

[[outputs.file]]

    files = ["C:/Program Files/Telegraf/metrics_output.txt","stdout"]
    data_format = "influx"

[[inputs.logparser]]

files = ["C:/Bitnami/wampstack-5.6.27-0/apache2/logs/access.log"]
 from_beginning = false
 [inputs.logparser.tags]
  intance = "apachetest80"

 [inputs.logparser.grok]
  patterns = ["%{COMBINED_LOG_FORMAT_WITH_TIME}"]
  measurement = "apache_access_log"
  custom_patterns = '''
  COMMON_LOG_FORMAT2 %{CLIENT:client_ip:drop} %{NOTSPACE:ident:drop} %{NOTSPACE:auth:drop} \[%{HTTPDATE:ts:ts-httpd}\] "(?:%{WORD:verb:drop} %{NOTSPACE:request:drop}(?: HTTP/%{NUMBER:http_version:drop})?|%{DATA})" %{NUMBER:resp_code:tag} (?:%{NUMBER:resp_bytes:drop}|-)
  COMBINED_LOG_FORMAT2 %{COMMON_LOG_FORMAT2} %{QS:referrer:drop} %{QS:agent:drop}
  COMBINED_LOG_FORMAT_WITH_TIME %{COMBINED_LOG_FORMAT2} %{INT:resptime:int}
    '''

[[aggregators.minmax]]
    period = "30s"        # send & clear the aggregate every 30s.
    drop_original = false  # drop the original metrics.
    namepass = ["apache_access_log"]

System info:

windows 2008 R2 //(telegraf-1.0.1_windows_amd64.zip)

[Include Telegraf version, operating system name, and other relevant details]

Steps to reproduce:

  1. C:\"Program Files"\Telegraf\telegraf.exe --config C:\"Program Files"\Telegraf\telegraf.conf --test
toni-moreno commented 8 years ago

I've tested also on linux with rpm telegraf-1.0.1.x86_64.rpm, and is also exiting with the same error.

# telegraf -config ./test_telegraf.conf
Error parsing ./test_telegraf.conf, Undefined but requested input: aggregators

I've copied config example from https://github.com/influxdata/telegraf/blob/master/docs/CONFIGURATION.md#aggregator-configuration-examples

sparrc commented 8 years ago

sorry for the confusion but that feature has been added in 1.1, it's not available yet

sparrc commented 8 years ago

@toni-moreno a release candidate for 1.1 should be available on the downloads page soon: https://www.influxdata.com/downloads/

toni-moreno commented 8 years ago

Hi @sparrc but I'm a bit confused over current versions, I'm compiling from sources rigth now commit (47de43abf3864b99465b5b033f420ce3d6e25e29) , and when building the executable version 1.0.0-beta3-209-g47de43a appears 

Administrator@WIN-PIUVQOCD78A MINGW64 ~/Desktop/workspace/src/github.com/influxdata/telegraf (master)
$ make build-windows
GOOS=windows GOARCH=amd64 go build -o telegraf.exe -ldflags \
                "-X main.version=1.0.0-beta3-209-g47de43a -X main.commit=47de43a -X main.branch=master" \
                ./cmd/telegraf/telegraf.go

There is a bug on the version number perhaps?

toni-moreno commented 8 years ago

Hi @sparrc sorry for the error related to the versions.

When testing with my recently compiled version (right now) I can not see any <fieldname>_max , <fieldname>_min on my output file. I think the aggregator is not running propertly.

Perhaps you would like to open the issue again...

NOTE: I'm inserting each second a apache formated line to the simulated access.log to test the aggregator, and below you can see the ouput file

apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895000000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895001000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895002000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895003000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895004000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895005000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895006000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895007000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895008000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895009000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895010000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895011000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895012000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895013000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895014000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895015000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895016000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895017000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895018000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895019000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895020000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895021000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895022000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895023000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895024000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895025000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895026000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895027000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895028000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895029000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895030000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895031000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895032000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895033000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895034000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895035000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895036000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895037000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895038000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895039000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895040000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895041000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895042000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895043000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895044000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895045000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895046000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895047000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895048000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895049000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895050000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895051000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895052000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895053000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895054000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895055000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895056000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895057000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895058000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895059000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895060000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895061000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895062000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895063000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895064000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895065000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895066000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895067000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895068000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895069000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895070000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895071000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895072000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895073000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895074000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895075000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895076000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895077000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895078000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895079000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895080000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895081000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895082000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895083000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895084000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895085000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895086000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895087000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895088000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895089000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895090000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895091000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895092000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895093000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895094000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895095000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895096000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895097000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895098000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895099000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895100000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895101000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895102000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895103000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895104000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895105000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895106000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895107000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895108000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895109000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895110000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895111000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895112000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895113000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895114000000
apache_access_log,host=WIN-PIUVQOCD78A,intance=apachetest80,resp_code=200 resptime=10i 1477643895115000000
sparrc commented 8 years ago

sure, if that's the case it's definitely a bug

sparrc commented 8 years ago

could you provide a sample log file?

toni-moreno commented 8 years ago

I've configured with debug = true, and no logfile . And here the output

C:\Program Files\Telegraf>C:\"Program Files"\Telegraf\telegraf.exe --config C:\"Program Files"\Telegraf\telegraf.conf --debug
2016/11/03 23:15:27 I! Using config file: C:\Program Files\Telegraf\telegraf.conf
2016-11-03T22:15:27Z D! Attempting connection to output: file
2016-11-03T22:15:27Z D! Successfully connected to output: file
2016-11-03T22:15:27Z I! Starting Telegraf (version 1.0.0-beta3-209-g47de43a)
2016-11-03T22:15:27Z I! Loaded outputs: file
2016-11-03T22:15:27Z I! Loaded inputs: inputs.logparser
2016-11-03T22:15:27Z I! Tags enabled: host=WIN-PIUVQOCD78A
2016-11-03T22:15:27Z I! Agent Config: Interval:10s, Quiet:false, Hostname:"WIN-PIUVQOCD78A", Flush Interval:1m0s
2016/11/03 23:15:27 Seeked C:/Bitnami/wampstack-5.6.27-0/apache2/logs/access.log - &{Offset:0 Whence:2}
2016-11-03T22:15:30Z D! Input [inputs.logparser] gathered metrics, (10s interval) in 0s
2016-11-03T22:15:40Z D! Input [inputs.logparser] gathered metrics, (10s interval) in 0s
2016-11-03T22:15:49Z D! Input [inputs.logparser] gathered metrics, (10s interval) in 0s
2016-11-03T22:15:58Z D! Input [inputs.logparser] gathered metrics, (10s interval) in 0s
2016-11-03T22:16:08Z D! Input [inputs.logparser] gathered metrics, (10s interval) in 0s
2016-11-03T22:16:19Z D! Input [inputs.logparser] gathered metrics, (10s interval) in 0s
2016-11-03T22:16:29Z D! Input [inputs.logparser] gathered metrics, (10s interval) in 0s
2016-11-03T22:16:29Z I! Output [file] buffer fullness: 59 / 1000 metrics. Total gathered metrics: 59. Total dropped metrics: 0.
2016-11-03T22:16:29Z I! Output [file] wrote batch of 59 metrics in 0s
2016-11-03T22:16:39Z D! Input [inputs.logparser] gathered metrics, (10s interval) in 0s
2016-11-03T22:16:49Z D! Input [inputs.logparser] gathered metrics, (10s interval) in 0s
2016-11-03T22:17:00Z D! Input [inputs.logparser] gathered metrics, (10s interval) in 0s
2016-11-03T22:17:14Z D! Input [inputs.logparser] gathered metrics, (10s interval) in 0s
2016-11-03T22:17:19Z D! Input [inputs.logparser] gathered metrics, (10s interval) in 0s
2016-11-03T22:17:29Z D! Input [inputs.logparser] gathered metrics, (10s interval) in 0s
2016-11-03T22:17:29Z I! Output [file] buffer fullness: 41 / 1000 metrics. Total gathered metrics: 100. Total dropped metrics: 0.
2016-11-03T22:17:29Z I! Output [file] wrote batch of 41 metrics in 0s
2016-11-03T22:17:39Z D! Input [inputs.logparser] gathered metrics, (10s interval) in 0s
2016-11-03T22:17:49Z D! Input [inputs.logparser] gathered metrics, (10s interval) in 0s
2016-11-03T22:17:59Z D! Input [inputs.logparser] gathered metrics, (10s interval) in 0s
sparrc commented 8 years ago

I believe this is actually a documentation issue. Unfortunately at the moment we do not support historical data in the aggregator plugins.

This is a bit of a thorny problem as we have to create a sliding window of cached metrics for each period. Fully supporting historical data would require creating an arbitrary number of windows as metrics arrive to the aggregator because we can't pre-determine the start/end time of each window.

One "easy" workaround might be to create a single "historical window" which would have it's period started on the timestamp of the first "historical" metric that arrives. This would cover your use-case where you have a single stream of historical and ordered metrics arriving (ordered is the crucial point). I imagine that when dealing with historical data, this would be the most common use-case, especially when dealing with logfile parsing.

sparrc commented 8 years ago

I created a separate issue for supporting historical data, see https://github.com/influxdata/telegraf/issues/1992

toni-moreno commented 8 years ago

Sorry but I'm a bit confused about how aggregator plugins gets data and process them.

On my use case I'm simulating 1 log line per second, and I have aggregation configured with period 10 seconds.

[[aggregators.minmax]]
    period = "10s"        # send & clear the aggregate every 10s.
    drop_original = false  # drop the original metrics.
    namepass = ["apache_access_log"]

In this scenario as yourself has added in https://github.com/influxdata/telegraf/commit/a5c8328b3255b9899a476227efc3464045d33970 , "In other words, if your metric timestamp is more than now() - period in the past, it will not be aggregated" , I expect 10 measurements added to the cache of the aggregator to be processed.

You can see in the ouput apache_access_log measurements (https://github.com/influxdata/telegraf/issues/1989#issuecomment-258274937) there is one measurement each second in the output.

IMHO, there is no historical data here, I expect only 10 measurements . Here the order is not important as max,min,mean,count,stdev not depends on the sample order.

There is perhaps a measurement buffer between input and aggregator blocking metrics for more than 10 seconds perhaps? If so , why ?

I'm also surprised because I can not understand another use case for the aggregator plugin different what statsd does with input data from logs , or any other with async generated data ( I'm sorry for my poor imagination here)

sparrc commented 8 years ago

When the aggregators start up they get a starting timestamp. If metrics are outside this timestamp then they are not aggregated.

If you take a look at the input plugins for telegraf, almost none of them accept data with a timestamp, the logparser is actually an outlier among them.

The aggregators basically assume that the data being aggregated is getting collected "now" (and has a timestamp "now". In your case, the logfile metrics have timestamps, meaning they are in the past and this is what I meant when I said "historical".

toni-moreno commented 8 years ago

Ok , I understand what you mean when you say "historical" , but I can not understand yet why aggregator is discarding measurements with correct timestamps in the range ( now- period ). what is the reason?

In the example above the only way aggregator gets timestamps outside the range (now-period) is a internal measurement buffer has been retaining a while metrics between input and aggregator blocking metrics for more than 10 seconds.

But there is no need for buffering here isn't it? sorry I'm still confused ...

sparrc commented 8 years ago

@toni-moreno there is no internal buffering at play, your logparser pattern is setting the timestamp of the metrics, specifically, it's this part: \[%{HTTPDATE:ts:ts-httpd}\]

I could be missing something as well but I'm assuming you are parsing a logfile with timestamps in the past.....the example output that you provided has timestamps from over a week ago.

could you provide the logfile? (the one you are parsing, not the telegraf log)

toni-moreno commented 8 years ago

Hi @sparrc you are right !! I can now see aggregator values when I've configured logparser to remove log formatted time \[%{HTTPDATE:ts:drop}\] .

I'm sorry , a different timestamp on each output make me think that aggregator was generating timestamp at parsing time. But after your review I can see why aggregator was discarding metrics.

Again sorry and thanks a lot