Closed Tri0L closed 6 years ago
Versions:
source 'https://rubygems.org' gem 'fluentd', '<=1.2.5' gem 'activesupport', '~>5.2.1' gem 'fluent-plugin-kubernetes_metadata_filter', '~>2.0.0' gem 'fluent-plugin-elasticsearch', '~>2.11.5' gem 'fluent-plugin-systemd', '~>1.0.1' gem 'fluent-plugin-detect-exceptions', '~>0.0.11' gem 'fluent-plugin-prometheus', '~>1.0.1' gem 'fluent-plugin-multi-format-parser', '~>1.0.0' gem 'fluent-plugin-grok-parser','~>2.2.0' gem 'oj', '~>3.6.5'
My config:
<source> @id fluentd-containers.log @type tail path /mnt/logs/*.log pos_file /var/log/es-containers.log.pos time_format %Y-%m-%dT%H:%M:%S.%NZ tag raw.kubernetes.* read_from_head true <parse> @type multi_format <pattern> format json time_key time time_format %Y-%m-%dT%H:%M:%S.%NZ </pattern> <pattern> format /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/ time_format %Y-%m-%dT%H:%M:%S.%N%:z </pattern> </parse> </source> <filter raw.kubernetes.**> @type parser key_name log keep_time_key true <parse> @type grok <grok> pattern %{IPORHOST} - \[%{IPORHOST:the_real_ip}\] - (?:-|%{USERNAME:remote_user}) \[%{HTTPDATE:time_local}\] "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) "(?:-|%{DATA:referer})" "(?:-|%{DATA:agent})" %{NUMBER:request_length} %{NUMBER:request_time} \[%{IPORHOST:proxy_upstream_name}\] %{IPORHOST:upstream_addr}:%{POSINT} %{NUMBER:upstream_response_length} %{NUMBER:upstream_response_time} %{NUMBER:upstream_status} %{BASE16NUM:req_id} </grok> <grok> pattern (?<timestamp>%{YEAR}[./]%{MONTHNUM}[./]%{MONTHDAY} %{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:pid}#%{NUMBER}: %{GREEDYDATA:errormessage} </grok> </parse> </filter> <match **> @type stdout @id stdout_output </match>
I'm trying to parse string like:
{"log":"127.0.0.1 - [127.0.0.1] - - [09/Sep/2018:12:13:28 +0000] \"POST /images/rpc HTTP/1.1\" 200 2724 \"-\" \"curl/7.54.0\" 515 0.007 [kube-public-my-service-80] 127.0.0.1:1000 2724 0.008 200 d01a314ea75b826dd35aabc40771b786\n","stream":"stdout","time":"2018-09-09T12:13:28.802648471Z"}
And after <filter> section, messages lose time field. If I delete <filter> config block, all working fine.
<filter>
time
Try v2.3.1 please and keep_time_key parameter.
keep_time_key
Versions:
My config:
I'm trying to parse string like:
And after
<filter>
section, messages losetime
field. If I delete<filter>
config block, all working fine.