open-telemetry / opentelemetry-operator

Kubernetes Operator for OpenTelemetry Collector
Apache License 2.0
1.16k stars 414 forks source link

Multi Traces same service name causes confusion #1280

Closed Kaitou786 closed 1 year ago

Kaitou786 commented 1 year ago

Hi, I have some services running in multiple namespaces with different versions. I am collecting the traces and sending it to the jaeger backend.

For example, I have a deployment name api in namespaceA and in namespaceB

I have specific instrumentation object in both namespace and in both of them I have specified

resource:
    addK8sUIDAttributes: true
    resourceAttributes:
      namespace: namespace

But in the jaeger UI there are traces merged and I see a single service there api which makes it challenging to know which trace is coming from which deployment.

is there any way to segregate these traces based on the namespace ?

pavolloffay commented 1 year ago

The instrumentation should collect the namespace resource attribute that can be used to filter the data.

The service name attribute could be perhaps overridden here https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1alpha1/instrumentation_types.go#L69 or via the env option

pavolloffay commented 1 year ago
Kaitou786 commented 1 year ago

Okay... I am already sending in the resourceAttributes from the Instrumentation object and I can see the namespace attribute being present on the traces, I think the problem is on the Jaegar UI side which lacks the filtering of these traces based on the attributed present But thanks for your response!

P.S. I have also tried to send in the service.name in the resource attributes but it doesn't seem like respect them either

Edit: I have tried sending service.name in the resourceAttributes but still it doesn't get respected... at least on the jaegar UI

I also tried with OTEL_SERVICE_NAME env var which works.

Kaitou786 commented 1 year ago

Closing since I am able to use env var to override it; though it would have been nice if the resourceAttributes were getting respected for service.name. Though I think that is a different issue.