Custom Metric Tags

[tellybrown]

I am trying to figure out how to get custom metric information from jaeger -> prometheus -> grafana. The data is flowing through but it is limited to basic information like a span was saved for my service. I would think something simple like the span name would be included.

I want this data to include more information that I specify. I am probably missing something. Maybe I have to send my own metrics but I cannot find anything useful searching "jaeger custom metrics".

jaeger_collector_traces_received_total{debug="false",format="jaeger",instance="localhost:16686",job="prometheus",svc="test.service"}

to something like this: jaeger_collector_traces_received_total{debug="false",format="jaeger",instance="localhost:16686",job="prometheus",svc="test.service",mytag="myvalue"}

I see on this class that I can add tags to a metric but that is a static class which means it cant be related to a specific span but rather all spans.

https://github.com/jaegertracing/jaeger-client-csharp/blob/master/src/Jaeger/Metrics/MetricsImpl.cs AddTagsToMetricName(string name, Dictionary<string, string> tags)

[Falco20019]

Jaeger itself is only generating the metrics shown in https://github.com/jaegertracing/jaeger-client-csharp/blob/master/src/Jaeger/Metrics/IMetrics.cs

But you can create all your custom metrics in Prometheus yourself independent of Jaeger or OpenTracing. The currently planned implementation looks like this: https://github.com/jaegertracing/jaeger-client-csharp/blob/Falco20019/prometheus/src/Jaeger.Prometheus/README.md

You can already use it by copying the PrometheusMetricsFactory.cs to your code case and adding it using the Tracer.Builder or Configuration using WithMetricsFactory as in the README.

This will add all tags as labels to the metric which might already solve part of what you are looking for. The package is currently not released since I wanted to wait for #116 to be merged (see #115).

[tellybrown]

It seems strange that I would need to make a second call with something I already am sending. Wouldn't it be best just to change the metrics endpoint in jaeger to add essentially (sql) a group by statement of all the custom tags?

http://localhost:16686/metrics

# TYPE jaeger_collector_traces_saved_by_svc_total counter
jaeger_collector_traces_saved_by_svc_total{debug="false",result="ok",svc="test.service"} 5

to something like

# TYPE jaeger_collector_traces_saved_by_svc_total counter
jaeger_collector_traces_saved_by_svc_total{debug="false",result="ok",svc="test.service",mytag="myvalue1"} 3
jaeger_collector_traces_saved_by_svc_total{debug="false",result="ok",svc="test.service",mytag="myvalue2"} 2

[Falco20019]

Are you using the PrometheusMetricsFactory as suggested?

[tellybrown]

Not yet. Just got back and researching what you posted. I will give it a try.

[Falco20019]

Sorry I misread the initial request since you talked about AddTagsToMetricName. This is only used for creating the full metric name and not for use by the end user. It’s public since Jaeger.Prometheus will also use it.

On what tags what you like to filter? The ones in the active span? Have you looked at other Jaeger languages? I think this is not available in any, so this is something we might want to discuss in the meta repository. I will check tomorrow the other implementations.

[yurishkuro]

@tellybrown is your question about metrics emitted from Jaeger backend components?

[tellybrown]

@yurishkuro I am new to all of this so you all will have to forgive me if I am not using this system as intended. What I plan on doing is adding the opentracing to my microservices to essentially log and create performance metrics across the system. We would use the jaeger ui for day to day troubleshooting. The prometheus -> grafana would be used to identify performance issues for each sub system as well as performance per tenant. I think this "per tenant" is what I am trying to make happen.

I can see my management wanting reports on how a particular tenant is doing during their peak periods.

[tellybrown]

@Falco20019 I don't think that Factory is the answer. While it would solve the problem, I don't particularly want to send the same/similar information twice. Once to jaeger and once to prometheus. I do think that solution is good for additional metrics on top of what you are already capturing so maybe in the future I might want it. Especially for back end job services.

[yurishkuro]

Jaeger backend is not intended to be the source of application metrics. The best you can do is reuse tracing instrumentation to emit something like the RED metrics. Unfortunately, the OpenTracing API does not currently provide Span Observer API, so Jaeger Go client implements an internal module for RPC metrics. There's also a decorator-based metrics generator for Java tracers, you can see it in action in https://github.com/PacktPublishing/Mastering-Distributed-Tracing/blob/master/Chapter11.

[tellybrown]

Thanks. I think I see your point now. Closing.