cosmo0920 / fluent-bit-go-loki

[Deprecated] The predessor of fluent-bit output plugin for Loki. https://github.com/grafana/loki
Apache License 2.0
40 stars 6 forks source link

Parsed log fields are ignored #17

Open Skeen opened 5 years ago

Skeen commented 5 years ago

Problem

Parsed log fields are ignored, and raw logs are written instead.

Steps to replicate

I have the following fluentbit configuration:

[SERVICE]
    Flush        1
    Daemon       Off
    Log_Level    warning
    HTTP_Server  true
    Parsers_File fluentbit/conf/parsers.conf

[INPUT]
    Name tail
    Tag logfiles.data
    Path /var/log/auth.log
    Path_Key path
    Key_Name log
    Parser auth

[OUTPUT]
    Name stdout
    Match *

[OUTPUT]
    Name  Loki
    Match *
    Url http://loki:3100/api/prom/push

Contents of fluentbit/conf/parsers.conf:

[PARSER]
    Name auth
    Format regex
    Regex ^(?<time>[^ ]* \d{2} \d{2}:\d{2}:\d{2}) (?<host>[\S]*) (?<program>[^\[\] ]*)(\[(?<pid>\d*)\])?: (?<message>.*)$
    Time_Key time
    Time_Format %b %d %H:%M:%S
    Time_Keep true

Contents of /var/log/auth.log:

Aug 22 13:26:45 plutonium systemd-logind[814]: Removed session 6422.

Expected Behavior or What you need to ask

I expected loki to store the stringified parsed data, similar to what stdout gives:

Other outputs, behave like this:

Using Fluentd and loki plugin versions

Ubuntu 16.04.6 LTS

Within docker, started with, command:

/fluent-bit/bin/fluent-bit -c /fluent-bit/etc/fluent-bit-loki.conf -e /usr/lib/x86_64-linux-gnu/out_loki.so

And these folder binds:

"/fluentbit/conf/fluent-bit.conf:/fluent-bit/etc/fluent-bit-loki.conf:rw",
"/fluentbit/conf/:/fluentbit/conf/:rw",
"/var/log/:/var/log/:rw"

Fluent Bit v1.1.3

level=info caller=out_loki.go:86 [flb-go]="Starting fluent-bit-go-loki" version="(version=0.2.0, branch=master, revision=5b25616)" level=info caller=out_loki.go:87 [flb-go]="provided parameter" URL=http://loki:3100/api/prom/push level=info caller=out_loki.go:88 [flb-go]="provided parameter" BatchWait= level=info caller=out_loki.go:89 [flb-go]="provided parameter" BatchSize= level=info caller=out_loki.go:90 [flb-go]="provided parameter" Labels= level=info caller=out_loki.go:91 [flb-go]="provided parameter" LogLevel= level=info caller=out_loki.go:92 [flb-go]="provided parameter" RemoveKeys=


* Grafana and Loki version

Grafana v6.4.0-pre (299a0e2) loki_build_info{branch="master",goversion="go1.11.4",revision="481906c4",version="master-481906c"} 1.0

cosmo0920 commented 5 years ago

This fluent-bit-go-loki plugin is merged into loki upstream. Could you test it instead? https://github.com/grafana/loki/tree/master/cmd/fluent-bit