corentinaltepe / nlog.loki

NLog target for Loki using an HTTP client
BSD 3-Clause "New" or "Revised" License
22 stars 7 forks source link

[Feature Request] Parse Properties as labels #75

Closed 240026763 closed 1 year ago

240026763 commented 1 year ago

I use properties in my log events which I want to put into Loki as Metadata (labels) but not every log event uses every property and I'd rather not have to statically define each property in the config. Can we add an option in the target to convert all the properties of the nlog event into Loki labels (perhaps could parse out ${all-event-properties:separator=|}? I use scoped, event, and global properties. I'm fine with the global properties being added into the target config manually since they are static once set (also makes a much more standardized way of pulling in things like app name without statically putting it in the file like ${gdc:item=AppName}

corentinaltepe commented 1 year ago

Hi @240026763 I'm not sure I want to implement such feature into the library because it goes against Loki's best practices : Static labels are good, Use dynamic labels sparingly

Are you sure you want to go this way?

It is generally much better to parse your loki logs into json to get the property you want to obtain as a label. Below an example loki query, where you filter logs by two Static labels, and one nlog property that is part of the json payload of the log content.

{label1="a", label2="b"} 
| json
| property1="c" 

Would that work for you?

240026763 commented 1 year ago

Hmm, I'll need to investigate it more to see if this is the direction I want to go. Thank you for the article.

corentinaltepe commented 1 year ago

Hi @240026763,

While using the template project today, I realized that the layout documented in the readme did not allow Loki to parse messages as JSON. I have modified the default layout in the readme such that the recommended layout is a JsonLayout. Please also have a look at the added section https://github.com/corentinaltepe/nlog.loki#json-layout--loki-query as I believe it should address your requirements. This way, all event properties are structured in messages, allowing Loki to query and filter on them, but they do not define streams in Loki.

Finally, if JSON is not your favorite way to serialize logs, Loki supports custom patterns. You could use NLog's CSV Layout for a customized, yet Loki parseable structure.

corentinaltepe commented 1 year ago

Hi again @240026763. Since this feature was requested again, it was made available in v2.1.0. You'll need to enable the feature eventPropertiesAsLabels. Please see the release notes and the readme for details https://github.com/corentinaltepe/nlog.loki/releases/tag/2.1.0