new feature: need exemplar support

[bigfish49]

As prometheus and grafana has supported exemplar feature, can mtail support this feature? here is some doc related: https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md https://grafana.com/docs/grafana/latest/basics/exemplars/ https://prometheus.io/docs/prometheus/latest/feature_flags/#exemplars-storage thanks!

[jaqx0r]

Oh sweet, yes we can. I've been wondering when they'd get around to it.

Do you have an example use in mind? I could attach the entire log line but perhaps we want to only add a substring?

On Wed, 28 Sept 2022, 23:08 bigfish49, @.***> wrote:

As prometheus and grafana has supported exemplar feature, can mtail support this feature? here is some doc related:

https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md https://grafana.com/docs/grafana/latest/basics/exemplars/

https://prometheus.io/docs/prometheus/latest/feature_flags/#exemplars-storage thanks!

— Reply to this email directly, view it on GitHub https://github.com/google/mtail/issues/684, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAXFX63YY7DDLIWH25N2YB3WAQ7N3ANCNFSM6AAAAAAQXYOPEA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[bigfish49]

In our use case, we need to attach substrings like trace-id, as we just want to link to the log context by using grafana. So, i think we should support this by adding a substring. I suppose we could have done this in this way:

1. define some prefix like "exemplarlabel", "exemplarvalue", "exemplartimestamp",
2. when we find such variable in a pattern, we auto update exemplar using those values
3. e.g. the rule is written like this: counter my_counter /tid=(?P[0-9A-Za-z]+)/ { my_counter++ } the output metrics should be like this: my_counter 11 # {trace_id="KOO5S4vxi0o"} 0

in the example above, exemplar value is not define, so just default it to 0.