Open jmacd opened 3 months ago
Pinging code owners:
connector/spanmetrics: @portertech @Frapschen
See Adding Labels via Comments if you do not have permissions to add labels yourself.
This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers
. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.
Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself.
Component(s)
connector/spanmetrics
Is your feature request related to a problem? Please describe.
OTEP 235 describes how to encode sampling probability, and now probabilistic sampler processor supports it.
I propose to two new boolean flags to the
Config
of this component:sampling_adjustment
(default: false) When disabled, each span counts 1. When enabled and sampling has been recorded, each span counts as the inverse of its sampling probability.fractional_counting
(default: false) When disabled, spans are counted as integer data points. When enabled, spans are counted as floating point number data points. _Note this only applies to Sum points, not to Histogram point count fields, because OpenTelemetry does not (currently) support floating-point count histograms.When the sampling adjustment feature is enabled and the fractional counting feature is disabled, there is a potential for errors to be introduced stemming from either inadequate precision or from the use of non-integer-reciprocal sampling probabilities.
As an example of the first case:
The sampler is configured with 33.33% sampling, which is sufficiently close to 1-in-3 that integer counts will have very small error using the threshold calculated by
pkg/sampling
. However, the sampler is also configured withsampling_precision: 1
which forces the effective probability down in this case. Note the rejection thresholdot=th:a
equals 10/16 = 37.5%, and the rejection thresholdot=th:b
equals 11/16 = 31.25%. The sampler will outputot=th:b
in this case, and the effective adjustment equals exactly1/(1 - 11/16)
= 3.2, which rounds down to 3 for a error of 6.7%. The user should raise sampling precision to lower the systematic error.As an example of the second case:
The sampler is configured for 75% sampling. This is exactly expressed using powers-of-two, and the adjustment in this case is 1.333. No amount of precision will help in this case. The user should choose sampling probabilities that equate with integer counts. This rules out sampling percentages above 50%.
Describe the solution you'd like
When a sampling adjustment is used without fractional counting, a warning will be issued for spans with sampling probability with an unacceptable margin of error.
Describe alternatives you've considered
When a sampling adjustment is used without fractional counting, a floating-point valued metric named
M_residue
will be incremented (for metric namedM
) with the residual error. This amount can be monitored and used to correct the integer-valued metric.Additional context
https://github.com/open-telemetry/semantic-conventions/pull/793