Open shinkle-procore opened 1 month ago
Pinging code owners:
processor/probabilisticsampler: @jpkrohling @jmacd
See Adding Labels via Comments if you do not have permissions to add labels yourself.
@shinkle-procore, would you like to add it yourself? It should be relatively easy to add this.
@jpkrohling If I had more bandwidth and experience with the stack, yes :) unfortunately it's not something I can work on now.
@jpkrohling I'm interested in working on this! Would you be able to guide me on what needs to be done?
would the correct place to add this metric be at https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/probabilisticsamplerprocessor/tracesprocessor.go#L174?
Close: I think this would be the right place: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/probabilisticsamplerprocessor/sampler_mode.go#L522
Instead of adding "1" to the new metric, it would add the count of items in the item T.
Component(s)
processor/probabilisticsampler
Is your feature request related to a problem? Please describe.
We are trying to build SLOs (Service Level Objectives) around our telemetry pipeline. One of the indicators we use is the ratio of "spans received by receiver" to "spans sent by the exporter" expecting that ratio to stay close to 1 (otherwise it means we're dropping spans during processing).
Now that we have probabilistic sampling set up (at 10%), we're finding that ratio is around 0.1 instead of 1, which makes sense. However, just adding a factor of 10 to our SLO measurement to compensate is hacky. We would like to factor something like "otelcol_processor_probabilistic_samplercountspans_sampled" into our calculation, but right now we only have access to "otelcol_processor_probabilistic_sampler_count_traces_sampled"
The telemetry in the core collector library seems to revolve around "spans" while the sampler metrics revolve around "traces", so it's difficult to compare the two in calculations.
Describe the solution you'd like
Emit a metric of "otelcol_processor_probabilistic_samplercountspans_sampled" from the probabilistic sampler processor.
Describe alternatives you've considered
No response
Additional context
No response