Support distinct count feature with OpenTelemetry metrics

[dpk83]

Feature Request

Is your feature request related to a problem? Yes.

If so, provide a concise description of the problem. Currently a lot of our services uses distinct count feature to setup their monitoring alerts. These services extensively uses the distinct count feature in backend to configure alerts to they only fire if a value of a specific metric tag is above certain threshold. Or to configure different severity based on the unique values of the specific dimension. To make the scenario more clear, here is a concrete example.

• Service cares about ensuring high reliability of a service and if something goes wrong they want to quickly detect the issue and mitigate it.
• Service owners configure alerts to fire whenever the overall reliability drops below, 99.9 %
• An alert in this case will generally fire at high severity and would require an on call engineer to engage immediately. This could be in the middle of night or anytime of the day.
• A misbehaving user or client who is hitting the service causes the reliability of the service to go below 99.9%. Or a corner case that's hitting a handful of users causes the reliability to go below 99.9% because these misbehaving clients might be making the failing requests in loop or any of the hundreds of other reasons.
• High severity alert gets fired and the on-call engineer needs to get up and engage immediately. He/she can then suppress the alert realizing it's not high sev because it's only hitting a very small number of users.
• As the misbehaving clients are still acting up, the alerts will fire again and the on call engineer need to engage again.

To avoid this situation and to assess the importance of an issue, services uses distinct count to determine the number of users impacted. The combination of unique number of users impacted and the reliability drop provides the right set of information to setup alerts at right severity.

With OpenTelemetry due to client side aggregation and the reason that such tags like userIds, or orgIds are high cardinality, the distinct count features is not supported.

Describe the solution you'd like:

Describe alternatives you've considered.

No alternatives other than staying on the existing non open telemetry metrics solution which provides the feature.

Additional Context

Add any other context about the feature request here.

[cijothomas]

With OpenTelemetry due to client side aggregation and the reason that such tags like userIds, or orgIds are high cardinality, the distinct count features is not supported.

This does not like something specific to OTel .NET but for all languages. The "distinct count" is not an aggregation part of the OTel sdk specification. I'd suggest to have this opened in specification repo to see if spec can add support for such thing, or open up pluggable custom aggregators.

[github-actions[bot]]

This issue was marked stale due to lack of activity and will be closed in 7 days. Commenting will instruct the bot to automatically remove the label. This bot runs once per day.

[github-actions[bot]]

Closed as inactive. Feel free to reopen if this issue is still a concern.