Can spanmetrics be exported to datadog?

[runtu666]

I know that spanmetrics can be exported to Prometheus, but after I switch to datadog, I can only see system metrics, can we export spanmetrics to datadog?

[songy23]

Hi @runtu666 can you share your collector config? You'll need a metrics pipeline that receives from spanmetrics and exports to datadog

[runtu666]

receivers: jaeger: protocols: thrift_http: endpoint: "0.0.0.0:14278"

otlp: protocols: grpc: http:

exporters: prometheus: endpoint: "0.0.0.0:8889"

jaeger: endpoint: "jaeger:14250" tls: insecure: true

datadog: api: site: "datadoghq.com" key: "xxxx"

connectors: spanmetrics:

processors: batch:

service: pipelines: traces: receivers: [otlp, jaeger] processors: [batch] exporters: [spanmetrics, jaeger]

The exporter name in this pipeline must match the spanmetrics.metrics_exporter name.

# The receiver is just a dummy and never used; added to pass validation requiring at least one receiver in a pipeline.
metrics/spanmetrics:
  receivers: [spanmetrics]
  exporters: [prometheus, datadog]

[runtu666]

@songy23 here is my config

[runtu666]

You'll need a metrics pipeline that receives from spanmetrics and exports to datadog

how can i do this ,can you show me an example?

[songy23]

Thanks @runtu666 for sharing your config. Config looks correct to me,

metrics/spanmetrics:
  receivers: [spanmetrics]
  exporters: [prometheus, datadog]

This is what I referred to as receiving from spanmetrics and exporting to datadog.

For our internal tracking, can you cut a support ticket https://help.datadoghq.com/hc/en-us/requests/new? BTW please don't share your Datadog API key publicly (you may want to rotate the API key in the comment).

[runtu666]

metrics/spanmetrics:
  receivers: [spanmetrics]
  exporters: [prometheus, datadog]

@songy23 but it doesn't working, I can only see system metrics not span metrics

[runtu666]

For our internal tracking, can you cut a support ticket https://help.datadoghq.com/hc/en-us/requests/new?

I can't open the link

[runtu666]

@songy23 if I use “metrics” pipeline ， I can see system metrics . if i use “metrics/spanmetrics” pipeline , I can see nothing

[songy23]

A few suggestions from @mx-psi :

1. Can you add a logging exporter to the metrics pipeline and get a sample output for metrics? e.g.

   
   exporters:
   logging:
   ....

metrics/spanmetrics: receivers: [spanmetrics] exporters: [prometheus, datadog, logging]

2. For the spanmetrics connector config, we recommend setting exponential histogram and delta aggregation_temporality. That will give a better experience when sending to Datadog:

connectors: spanmetrics: histogram: exponential: aggregation_temporality: AGGREGATION_TEMPORALITY_DELTA

[runtu666]

@songy23 It still doesn't working , and here is my config :

receivers:
  jaeger:
    protocols:
      thrift_http:
        endpoint: "0.0.0.0:14278"

  otlp:
    protocols:
      grpc:
      http:

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"

  jaeger:
    endpoint: "jaeger:14250"
    tls:
      insecure: true

  datadog:
    api:
      site: "datadoghq.com"
      key: "xxxxxxx"

  logging:

connectors:
  spanmetrics:
    histogram:
      exponential:
    aggregation_temporality: AGGREGATION_TEMPORALITY_DELTA

processors:
  batch:

service:
  pipelines:
    traces:
      receivers: [otlp, jaeger]
      processors: [batch]
      exporters: [spanmetrics, jaeger, datadog]
    # The exporter name in this pipeline must match the spanmetrics.metrics_exporter name.
    # The receiver is just a dummy and never used; added to pass validation requiring at least one receiver in a pipeline.
    metrics/spanmetrics:
      receivers: [spanmetrics]
      exporters: [prometheus, datadog, logging]

[runtu666]

[runtu666]

I have successfully sent the metrics to datadog. What do these metrics represent? Is there any document for reference? like 'calls duration_bucket'

@songy23

[songy23]

Glad to hear it's working for you. You can find the general information on span metrics in https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/spanmetricsconnector#overview. For Datadog metrics mapping, see https://docs.datadoghq.com/metrics/open_telemetry/otlp_metric_types/?tab=histogram#metric-types.

[karmingc]

If you don't mind me asking @songy23 on this similar topic, is there a way to configure the percentiles "advanced feature" through the exporter configuration? I looked, but can't seem to find anything apart from doing it manually on the UI...