search

open-telemetry / opentelemetry-python

OpenTelemetry Python API and SDK
https://opentelemetry.io
Apache License 2.0
1.71k stars 599 forks source link

Measure number of dropped spans in `BatchSpanProcessor` #3152

Open wmwmw opened 1 year ago

wmwmw commented 1 year ago

Is your feature request related to a problem? We would like to measure the number of dropped spans with the BatchSpanProcessor, simple True / False flag is not sufficient, as some (low) numbers of drops might not indicate anything bad.

Describe the solution you'd like Having a simple way to add a meter without having to override or wrap on_exit to see how many spans were dropped because of a full queue. Ideally, a call to a callback when spans are dropped due to a full queue.

Describe alternatives you've considered Re-implementing the on_exit of BatchSpanProcessor with added on_drop callback with a meter or adding a meter directly there. Eventually wrapping the existing method by checking the _spans_dropped flag and changing it back to false after incrementing a meter. Both methods might interfere with future changes, the latter would increase the log rate.

aabmass commented 1 year ago

Are you trying to access the counts programmatically or just monitor the queue? In the spirit of OTel, we should consider adding the metric instrumentation directly to the BatchSpanProcessor code.

wmwmw commented 1 year ago

I am just trying to monitor it, so doing it that way would also solve the issue for me and I don't need to have the number anywhere else.

sreejitkar commented 1 year ago

Hi , Is this issue still open to be picked up ? cc - @srikanthccv

lzchen commented 1 year ago

@sreejitkar

Would you like to be assigned?

sreejitkar commented 1 year ago

Yes please. Thanks !

rajat315315 commented 1 year ago

I have submitted a PR for this, Can someone please review and suggest improvements?

soumyadeepm04 commented 1 month ago

Was having a look at this issue and noticed that when adding spans to the BatchSpanProcessor's queue, its size is never greater than the max_export_batch_size. BatchSpanProcessor validates that the max_export_batch_size is always <= max_queue_size (see _validate_arguments function that throws an error when max_export_batch_size > max_queue_size). This means that it is not possible to have any dropped spans. Please correct me if I am wrong.