search

jaegertracing / jaeger

CNCF Jaeger, a Distributed Tracing Platform
https://www.jaegertracing.io/
Apache License 2.0
20.58k stars 2.45k forks source link

Build a Streaming Aggregation Processor For Service Map #6153

Open mahadzaryab1 opened 3 weeks ago

mahadzaryab1 commented 3 weeks ago

Implement a processor in the OTEL pipeline that takes as input a stream of traces and constructs a dependency graph between services.

mahadzaryab1 commented 3 weeks ago

@yurishkuro in this first step, do we just want to compute the dependencies in an apache beam pipeline?

yurishkuro commented 3 weeks ago

Yes but also implement that as an OTEL Processor. You can probably copy the existing processor from contrib (mentioned in the comments above) as a baseline.

mahadzaryab1 commented 3 weeks ago

@yurishkuro Are you referring the tailsamplingprocessor? https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor

mahadzaryab1 commented 3 weeks ago

@yurishkuro Another question - looks like OTEL has a groupbytraceprocessor. Is there any way we could leverage this or do we need to reuse this logic inside the processor that we write?

yurishkuro commented 3 weeks ago

No, I was referring to ServiceGraphConnector

groupbytraceprocessor is not needed for memory/badger, but may be needed for distributed storage. It somewhat depends on how Apache Beam implements the streams - if we have to shard spans by trace ID manually then something from OTEL will be needed (it's the same problem as with tail sampling processor, which needs full traces in one collector).