New component: processor/datadog

[gbbr]

The purpose and use-cases of the new component

Collects pre-sampling trace metrics. Users who wish to use the probabilistic sampler or the tailsamplingprocessor can prepend the "datadog" processor in their pipelines to see the full picture. Please see the example configuration below.

Example configuration for the component

Please note the new datadog processor addition.

receivers:
  otlp:
    protocols:
      http:

processors:
  batch:
  probabilistic_sampler:
    sampling_percentage: 10
  datadog:

exporters:
  datadog:
    api:
      key: ${DD_API_KEY}

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch, k8sattributes]
      exporters: [datadog]
    traces:
      receivers: [otlp]
      processors: [batch, k8sattributes, datadog, probabilistic_sampler]
      exporters: [datadog]

An exporter_name setting would also be available if one wishes to use a non-Datadog exporter. Otherwise, the datadog processor would automatically detect the presence of the Datadog exporter and use that.

Telemetry data types supported

Traces.

Is this a vendor-specific component?

• [X] This is a vendor-specific component
• [X] If this is a vendor-specific component, I am proposing to contribute this as a representative of the vendor.

Sponsor (optional)

@mx-psi

Additional context

Why don't you use the spanmetricsprocessor?

While this seems to be the most natural solution, there are some disadvantages and limitations that are blocking it:

• Stats accuracy is too low for Datadog. It uses Histogram instead of ExponentialHistogram.
• Impossible to obtain the original instrumentation scope which is overridden to "spanmetricsprocessor". This is needed because it is an essential property.
• Is high cardinality because it requires the collection of many dimensions to deduce the Datadog counter-part attributes.
• Is marked as experimental.
• Is meant for generic use cases instead of a specific vendor product.
• Does not obfuscate SQL queries, which is needed for aggregation keys to function correctly at Datadog.
• Hacky setup that needs to specify exporter vs. automatic detection that is being proposed here.
• Can not communicate to a potentially remote Datadog Exporter (in the case of gateway deployments) that stats have already been computed elsewhere and it should skip doing that. The proposed processor can just intercept the traces that pass through it and add a resource attribute to mark them as "computed".

Why don't you use two separate pipelines?

One could use two separate pipelines with two separate exporters:

• One for traces with sampling
• One for traces without sampling that exports only stats

This change involves adding a second exporter, which is fine (considering it is accepted by the community), but raises several difficulties:

• Memory usage is increased. Traces are duplicated in memory (because the batch processor marks itself as mutating).
• Config needs to be duplicated between the two exporters (API keys, etc), which makes it error prone. YAML anchors & aliases can mitigate this.
• Worse user experience in terms of configuration: the user not only has to duplicate the exporter config, but also the pipelines (minus the sampler), which is error prone.
• Does not work very well with gateway deployments. If a local collector has the Datadog trace metrics exporter, it can not communicate to the existing exporter to skip the stats computation, resulting in duplicate computation.

[evan-bradley]

@mx-psi Can you confirm that you will be sponsoring this component?

[mx-psi]

Yes, I will be sponsoring this, but we want to make sure that this is the right choice since we would be the first vendor to have a vendor-specific processor.

[gbbr]

Since there are no objections here, I'm going to start working on this.

[jpkrohling]

Since there are no objections here, I'm going to start working on this.

Was this discussed during a SIG meeting?

[mx-psi]

Since there are no objections here, I'm going to start working on this.

Was this discussed during a SIG meeting?

No, it was not. This component is vendor specific and I sponsored it, but if you have thoughts on the design or approach please comment then now better than later!

Note that it's possible that once connectors are a thing we may want to change this and the spanmetricsprocessor into connectors instead, but this seemed like the best approach at the time when we were discussing this.