search

cosmo0920 / fluent-bit-go-loki

[Deprecated] The predessor of fluent-bit output plugin for Loki. https://github.com/grafana/loki
Apache License 2.0
40 stars 6 forks source link

How to add the parsed log fields as Labels #9

Closed Vaka82 closed 5 years ago

Vaka82 commented 5 years ago

Problem

How to add the parsed log fields as Labels. So that i can search using those parsed fields in Grafana.

I've parsed the log lines using regex as below -

{"userid":"test",component"=>"test1", "page"=>"test2"}

I would like to add these labels and search using these labels in grafana.

cosmo0920 commented 5 years ago

There is no support this feature for now. And I don't want to add this feature....

tfchan0 commented 5 years ago

Thanks for this plugin! Along the same line, that means it's not supported to take existing labels (keys) in the logline as Loki labels at this time? Thanks.

Vaka82 commented 5 years ago

I've added that logic to label those required fields but that made the search terribly slow. Often times it was crashing.  Also whike loading the data it's throwing data out of order exceptions...looks like loki need those labels to be unique...

tfchan0 commented 5 years ago

I believe so. I followed the tutorial by adding worker_id/hostname (I'm sourcing from Fluentd). Then it's fine afterwards.

tfchan0 commented 5 years ago

@Vaka82, would you be able to share the added logic for parsing log fields as Label? Thanks.

Skeen commented 5 years ago

@Vaka82 I'd be interested in this as well, having the option to extract at least a few labels from the json would be massively helpful.

cosmo0920 commented 5 years ago

This feature will be included in v0.3.0. It is developed on https://github.com/cosmo0920/fluent-bit-go-loki/tree/support-dynamic-labels.

cosmo0920 commented 5 years ago

v0.3.0 is published. Closing.