Datadog exporter - Support for Histogram percentiles

[datsabk]

Component(s)

exporter/datadog

Is your feature request related to a problem? Please describe.

As per this document, Datadog exporter should be generated approximiate percentiles (Quoting the statement below). However, as of today, percentile generation is not supported in Datadog exporter - Count and Sum metrics do land as expected in Datadog.

Note: Histogram metrics in OTLP are mapped to Distribution metrics. Because of how OTLP sends this data, the max, min, and percentile aggregations are approximations, not accurate calculations.

Requesting expertise from Datadog team to successfully pipeline percentile metrics (similar to what DogStatsD server does)

Describe the solution you'd like

metrics:
      resource_attributes_as_tags: true
      histograms:
        send_aggregation_metrics: true
        percentiles: 
          send_percentile_metrics: true
          percentiles:
            - 95
            - 99
            - 75

This would allow us to generate percentiles as needed similar to Datadog agent.

Describe alternatives you've considered

The only other alternate is to get rid of Open Telemetry collector and start using Datadog agent for this purpose.

Additional context

No response

[github-actions[bot]]

Pinging code owners:

• exporter/datadog: @mx-psi @gbbr @dineshg13 @liustanley @songy23

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[datsabk]

Investigated further into the code - Looking at this it seems to be using some beta api for sketches which is no-where documented.

I wonder if the API no longer works - @mx-psi Any chance if you know about this piece of code or how it works ?

I am hoping to generate percentile metrics out of the Prometheus metric buckets similar to the ones shown below:

awesome_product_healthcheck_plugin_duration_seconds_bucket{le="0.005",plugin="catalog",probeType="startz"} 0
awesome_product_healthcheck_plugin_duration_seconds_bucket{le="0.01",plugin="catalog",probeType="startz"} 0
awesome_product_healthcheck_plugin_duration_seconds_bucket{le="0.025",plugin="catalog",probeType="startz"} 1
awesome_product_healthcheck_plugin_duration_seconds_bucket{le="0.05",plugin="catalog",probeType="startz"} 1
awesome_product_healthcheck_plugin_duration_seconds_bucket{le="0.1",plugin="catalog",probeType="startz"} 1
awesome_product_healthcheck_plugin_duration_seconds_bucket{le="0.25",plugin="catalog",probeType="startz"} 1
awesome_product_healthcheck_plugin_duration_seconds_bucket{le="0.5",plugin="catalog",probeType="startz"} 1
awesome_product_healthcheck_plugin_duration_seconds_bucket{le="1",plugin="catalog",probeType="startz"} 1
awesome_product_healthcheck_plugin_duration_seconds_bucket{le="2.5",plugin="catalog",probeType="startz"} 1
awesome_product_healthcheck_plugin_duration_seconds_bucket{le="5",plugin="catalog",probeType="startz"} 1
awesome_product_healthcheck_plugin_duration_seconds_bucket{le="10",plugin="catalog",probeType="startz"} 1
awesome_product_healthcheck_plugin_duration_seconds_bucket{le="+Inf",plugin="catalog",probeType="startz"} 1

[datsabk]

@mx-psi @dineshg13 - This might be a Bug. I see that sketches API is used for publishing to Datadog already. But the percentiles are not being published for some reason

Is there any documentation around sketches api? How does it work? Will it publish percentiles ?

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/datadog: @mx-psi @gbbr @dineshg13 @liustanley @songy23

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[mx-psi]

@mx-psi @dineshg13 - This might be a Bug. I see that sketches API is used for publishing to Datadog already. But the percentiles are not being published for some reason

Is there any documentation around sketches api? How does it work? Will it publish percentiles ?

@datsabk Apologies for the delay. Enabling percentiles is a feature that is configured in the backend, you can check out https://docs.datadoghq.com/metrics/distributions/#enabling-advanced-query-functionality

Since this is not specific to the Datadog exporter, I am going to close this as 'not applicable'. If you have further questions about how to work with percentiles on Datadog distributions/sketches please reach out to Datadog support instead. Thanks!