fluent / fluent-plugin-grok-parser

Fluentd's Grok parser
Other
107 stars 31 forks source link

Lose time field after grok filter #55

Closed Tri0L closed 6 years ago

Tri0L commented 6 years ago

Versions:

source 'https://rubygems.org'

gem 'fluentd', '<=1.2.5'
gem 'activesupport', '~>5.2.1'
gem 'fluent-plugin-kubernetes_metadata_filter', '~>2.0.0'
gem 'fluent-plugin-elasticsearch', '~>2.11.5'
gem 'fluent-plugin-systemd', '~>1.0.1'
gem 'fluent-plugin-detect-exceptions', '~>0.0.11'
gem 'fluent-plugin-prometheus', '~>1.0.1'
gem 'fluent-plugin-multi-format-parser', '~>1.0.0'
gem 'fluent-plugin-grok-parser','~>2.2.0'
gem 'oj', '~>3.6.5'

My config:

<source>
  @id fluentd-containers.log
  @type tail
  path /mnt/logs/*.log
  pos_file /var/log/es-containers.log.pos
  time_format %Y-%m-%dT%H:%M:%S.%NZ
  tag raw.kubernetes.*
  read_from_head true
  <parse>
    @type multi_format
    <pattern>
      format json
      time_key time
      time_format %Y-%m-%dT%H:%M:%S.%NZ
    </pattern>
    <pattern>
      format /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/
      time_format %Y-%m-%dT%H:%M:%S.%N%:z
    </pattern>
  </parse>
</source>

<filter raw.kubernetes.**>
  @type parser
  key_name log
  keep_time_key true
  <parse>
    @type grok
    <grok>
      pattern %{IPORHOST} - \[%{IPORHOST:the_real_ip}\] - (?:-|%{USERNAME:remote_user}) \[%{HTTPDATE:time_local}\] "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) "(?:-|%{DATA:referer})" "(?:-|%{DATA:agent})" %{NUMBER:request_length} %{NUMBER:request_time} \[%{IPORHOST:proxy_upstream_name}\] %{IPORHOST:upstream_addr}:%{POSINT} %{NUMBER:upstream_response_length} %{NUMBER:upstream_response_time} %{NUMBER:upstream_status} %{BASE16NUM:req_id}
    </grok>
    <grok>
      pattern (?<timestamp>%{YEAR}[./]%{MONTHNUM}[./]%{MONTHDAY} %{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:pid}#%{NUMBER}: %{GREEDYDATA:errormessage}
    </grok>
  </parse>  
</filter> 

<match **>
  @type stdout
  @id stdout_output
</match>

I'm trying to parse string like:

{"log":"127.0.0.1 - [127.0.0.1] - - [09/Sep/2018:12:13:28 +0000] \"POST /images/rpc HTTP/1.1\" 200 2724 \"-\" \"curl/7.54.0\" 515 0.007 [kube-public-my-service-80] 127.0.0.1:1000 2724 0.008 200 d01a314ea75b826dd35aabc40771b786\n","stream":"stdout","time":"2018-09-09T12:13:28.802648471Z"}

And after <filter> section, messages lose time field. If I delete <filter> config block, all working fine.

okkez commented 6 years ago

Try v2.3.1 please and keep_time_key parameter.