josephwoodward / Serilog-Sinks-Loki

A Serilog Sink for Loki, Grafana's new Prometheus inspired log aggregator
MIT License
136 stars 35 forks source link

Labels with invalid identifiers and multline text #19

Open ghost opened 4 years ago

ghost commented 4 years ago

Hello everyone and thanks for the great work!

I am facing a problem while using the sink (v2.1.0) with loki server (v1.5). When I log something the following exception happens:

2020-07-03T12:14:39.2871483Z Exception while emitting periodic batch from Serilog.Sinks.Http.Private.Sinks.HttpSink: Serilog.Debugging.LoggingFailedException: Received failed result BadRequest when posting events to http://localhost:3100/api/prom/push
   at Serilog.Sinks.Http.Private.Sinks.HttpSink.EmitBatchAsync(IEnumerable`1 logEvents) in C:\projects\serilog-sinks-http\src\Serilog.Sinks.Http\Sinks\Http\Private\Sinks\HttpSink.cs:line 94
   at Serilog.Sinks.PeriodicBatching.PeriodicBatchingSink.OnTick()

The loki server returns the following:

error parsing labels: parse error at line 1, col 34: literal not terminated

I use a library called fo-dicom, and when it logs something, it adds some properties that are problematic. For instance, one of the properties (which are turned into labes by Serilog-Sinks-Loki) contains a name that is an invalid identifier (fo-dicom which contains a -) and the other contains line breaks. When the following requests are sent, they return BadRequest (400):

{"streams":[{"labels":"{level=\"info\",app=\"demo\", label1=\"test \n multiline\", namespace=\"prod\",MyPropertyName=\"MyPropertyValue\",ThreadId=\"1\"}","entries":[{"ts":"2020-07-03T13:50:55.6289463-03:00","line":"Linha Teste\n"}]}]}

This one fails because label1 has a multiline text. Other issue I found is:

{"streams":[{"labels":"{level=\"info\",app=\"demo\", invalid-label=\"test \n multiline\", labelinvalid=\"teste\", namespace=\"prod\",MyPropertyName=\"MyPropertyValue\",ThreadId=\"1\"}","entries":[{"ts":"2020-07-03T13:50:55.6289463-03:00","line":"Linha Teste\n"}]}]}

Since "invalid-label" contains a -, loki server returns:
error parsing labels: parse error at line 1, col 32: syntax error: unexpected -, expecting = or != or =~ or !~

I believe it is necessary to remove any linebreaks from labels, and also make sure they property names are valid identifiers. I've done a workaround to make it work, but it would be nice if the sink itself handled these cases.

qiqistart commented 3 years ago

I’d like to know that how to use serilog.loki to establish a customizing client to make an index for writing logs as Elasticsearch.