search

jenkinsci / opentelemetry-plugin

Monitor and observe Jenkins with OpenTelemetry.
https://plugins.jenkins.io/opentelemetry/
Apache License 2.0
96 stars 50 forks source link

SERVICE_NAMESPACE_AND_NAME does not play nicely with Tempo tags #750

Closed chewrocca closed 10 months ago

chewrocca commented 10 months ago

What feature do you want to see added?

SERVICE_NAMESPACE_AND_NAME does not seem to play nicely with Tempo tags in Grafana Tempo Datasource

This line seems to build the Grafana Loki logs query:

https://github.com/jenkinsci/opentelemetry-plugin/blob/c03877572faca269d3691d1349976aebcf603ba9/src/main/java/io/jenkins/plugins/opentelemetry/backend/grafana/GrafanaLogsBackend.java#L66

I don't think we can have two keys in Tempo Datasource. In this case, service.name and service.namespace map to one value (that would be nice). I was trying to follow the OpenTelemetry Loki Exporter to drop service.namespace which should mean "If service.name is present and service.namespace is not present then job=service.name is set."

Without Tempo Datasource mapping two keys to one value, or allowing this Grafana Loki logs query to have only service.name it's making traces to logs correlations a bit challenging.

Upstream changes

No response

chewrocca commented 10 months ago

This issue seems related. https://github.com/grafana/loki/issues/4553

cyrille-leclerc commented 10 months ago

Please see new release v2.19.0, you can configure sending logs to Loki and have links to navigate to Grafana for visualization.

You will also see how we do querying and exploration of Loki logs.

Please note that the decision to produce the job and instance labels when mapping OTLP logs to Loki is not the decision of the Jenkins OpenTelemetry plugin, it's the overall mapping currently defined by the OTel Collector Loki Exporter that is going through many improvements at the moment. We acknowledge that having to parse OTel logs persisted in Loki using the json parsing is a form of boilerplate and the Loki project is working at simplifying this.

Please tell us how v2.19.0 helps you.

https://github.com/jenkinsci/opentelemetry-plugin/releases/tag/opentelemetry-2.19.0

chewrocca commented 10 months ago

I believe my real issue is with the tracesToLogsV2 tags and custom query within the Tempo datasource configuration. I'll open an issue with them. They need a way to match two tag keys, in this case service.namespace/service.name to one value called job.

cyrille-leclerc commented 10 months ago

Understood @chewrocca . Note that on Grafana Cloud, you can use Grafana Cloud Application Observability where we have solved this trace2logs question.

@mar4uk please see this interesting use cas where curent Loki labels for OTel logs makes it difficult to implement trace2logs with standard Grafana capabilities.

@chewrocca FYI @mar4uk is the Grafana Labs engineer who maintains the OTel Collector exporter for Loki, she is one of the best experts of the otel2loki mapping.