fangli / fluent-plugin-influxdb

A buffered output plugin for fluentd and InfluxDB
MIT License
111 stars 65 forks source link

Number/float field are pushed into influxdb as string #101

Closed autumnw closed 5 years ago

autumnw commented 5 years ago

Hi, I try to use this plugin to convert nignx access log into metrics and push it into influxdb. The nginx access log format likes the following:

[26/Oct/2019:17:21:40 +0000] 302 "-@127.0.0.1" "GET / HTTP/1.0" 4801 "-" "-" 0.631 0.631 .

I expect the status code(302), request time(0.631) and response time(0.631) should be saved into influxdb AS number/float data type. But actually all of them are stored as string.

> select * from fe_portal
name: fe_portal
time                agent        code hostname               method path project   referer region          remote    request_time response_time size tailed_path                     timestamp                  user version
----                -----        ---- --------               ------ ---- -------   ------- ------          ------    ------------ ------------- ---- -----------                     ---------                  ---- -------
1572110977000000000 GoogleHC/1.0 302  wf20-td-agent-test-1-0 GET    /    prod-test -       asia-southeast1 127.0.0.1 0.630        0.630         4801 /var/log/pan/my-access-test.log 26/Oct/2019:17:29:21 +0000 -    HTTP/1.0
> select request_time::float from fe_portal
> select request_time::string from fe_portal
name: fe_portal
time                request_time
----                ------------
1572110977000000000 0.630
>

My fluentd config likes the following:

<source>
  @type tail
  pos_file /var/log/td-agent/nginx-influx.pos
  path_key tailed_path
  @id input_tail
  path /var/log/pan/*access*.log
  tag fe_portal
  <parse>
    @type regexp
    expression /^\[(?<timestamp>[^\]]*)\] (?<code>[^ ]*) "(?<user>[^ ]*)@(?<remote>[^ ]*)" "(?<method>\S+)(?: +(?<path>[^\ "]*?)(?: +\S*)?) (?<version>\S+/\S+)" (?<size>[\d.]+) "(?<referer>[^ ]*)" "(?<agent>[^\"]*)" (?<request_time>[\d.-]+) (?<response_time>[\d.-]+).*$/
    time_format %d/%b/%Y:%H:%M:%S %z
  </parse>
</source>
<filter fe_portal>
  @type record_transformer
  auto_typecast true
  <record>
    hostname ${hostname}
    project prod-test
    region asia-southeast1
  </record>
</filter>
<match fe_portal>
  type influxdb
  num_threads 8
  dbname test
  host influxdb
  port 8086
  flush_interval 10s
  cast_number_to_float true
  @id influxdb
  tag_keys ["method","path", "code", "region", "hostname"]
</match>
autumnw commented 5 years ago

I got it. The type convert should be in fluentd configuration