Open Skeen opened 5 years ago
Parsed log fields are ignored, and raw logs are written instead.
I have the following fluentbit configuration:
[SERVICE] Flush 1 Daemon Off Log_Level warning HTTP_Server true Parsers_File fluentbit/conf/parsers.conf [INPUT] Name tail Tag logfiles.data Path /var/log/auth.log Path_Key path Key_Name log Parser auth [OUTPUT] Name stdout Match * [OUTPUT] Name Loki Match * Url http://loki:3100/api/prom/push
Contents of fluentbit/conf/parsers.conf:
fluentbit/conf/parsers.conf
[PARSER] Name auth Format regex Regex ^(?<time>[^ ]* \d{2} \d{2}:\d{2}:\d{2}) (?<host>[\S]*) (?<program>[^\[\] ]*)(\[(?<pid>\d*)\])?: (?<message>.*)$ Time_Key time Time_Format %b %d %H:%M:%S Time_Keep true
Contents of /var/log/auth.log:
/var/log/auth.log
Aug 22 13:26:45 plutonium systemd-logind[814]: Removed session 6422.
I expected loki to store the stringified parsed data, similar to what stdout gives:
logfiles.data: [1566480405.000000000, {"path"=>"/var/log/auth.log", "time"=>"Aug 22 13:26:45", "host"=>"plutonium", "program"=>"systemd-logind", "pid"=>"814", "message"=>"Removed session 6422."}]
{"log":"Aug 22 13:26:05 plutonium systemd-logind[814]: Removed session 6422.","path":"/var/log/auth.log"}
Other outputs, behave like this:
[{u'pid': u'814', u'host': u'plutonium', u'program': u'systemd-logind', u'time': u'Aug 22 13:26:45', u'date': 1566480405.0, u'path': u'/var/log/auth.log', u'message': u'Removed session 6422.'}]
logfiles.data: [1566480405.000000, {"path":"/var/log/auth.log", "time":"Aug 22 13:26:45", "host":"plutonium", "program":"systemd-logind", "pid":"814", "message":"Removed session 6422."}]
Ubuntu 16.04.6 LTS
Within docker, started with, command:
/fluent-bit/bin/fluent-bit -c /fluent-bit/etc/fluent-bit-loki.conf -e /usr/lib/x86_64-linux-gnu/out_loki.so
And these folder binds:
"/fluentbit/conf/fluent-bit.conf:/fluent-bit/etc/fluent-bit-loki.conf:rw", "/fluentbit/conf/:/fluentbit/conf/:rw", "/var/log/:/var/log/:rw"
Fluent Bit v1.1.3
Fluent Bit v1.1.3 Copyright (C) Treasure Data
level=info caller=out_loki.go:86 [flb-go]="Starting fluent-bit-go-loki" version="(version=0.2.0, branch=master, revision=5b25616)" level=info caller=out_loki.go:87 [flb-go]="provided parameter" URL=http://loki:3100/api/prom/push level=info caller=out_loki.go:88 [flb-go]="provided parameter" BatchWait= level=info caller=out_loki.go:89 [flb-go]="provided parameter" BatchSize= level=info caller=out_loki.go:90 [flb-go]="provided parameter" Labels= level=info caller=out_loki.go:91 [flb-go]="provided parameter" LogLevel= level=info caller=out_loki.go:92 [flb-go]="provided parameter" RemoveKeys=
* Grafana and Loki version
Grafana v6.4.0-pre (299a0e2) loki_build_info{branch="master",goversion="go1.11.4",revision="481906c4",version="master-481906c"} 1.0
This fluent-bit-go-loki plugin is merged into loki upstream. Could you test it instead? https://github.com/grafana/loki/tree/master/cmd/fluent-bit
Problem
Parsed log fields are ignored, and raw logs are written instead.
Steps to replicate
I have the following fluentbit configuration:
Contents of
fluentbit/conf/parsers.conf
:Contents of
/var/log/auth.log
:Expected Behavior or What you need to ask
I expected loki to store the stringified parsed data, similar to what stdout gives:
Other outputs, behave like this:
Using Fluentd and loki plugin versions
Ubuntu 16.04.6 LTS
Within docker, started with, command:
And these folder binds:
Fluent Bit v1.1.3
level=info caller=out_loki.go:86 [flb-go]="Starting fluent-bit-go-loki" version="(version=0.2.0, branch=master, revision=5b25616)" level=info caller=out_loki.go:87 [flb-go]="provided parameter" URL=http://loki:3100/api/prom/push level=info caller=out_loki.go:88 [flb-go]="provided parameter" BatchWait= level=info caller=out_loki.go:89 [flb-go]="provided parameter" BatchSize= level=info caller=out_loki.go:90 [flb-go]="provided parameter" Labels= level=info caller=out_loki.go:91 [flb-go]="provided parameter" LogLevel= level=info caller=out_loki.go:92 [flb-go]="provided parameter" RemoveKeys=
Grafana v6.4.0-pre (299a0e2) loki_build_info{branch="master",goversion="go1.11.4",revision="481906c4",version="master-481906c"} 1.0