Span service.name attribute ignored during NativeIngest APM stats metric aggregation

[tinyjacky]

Component(s)

connector/datadog

What happened?

Description

With connect.datadogconnector.NativeIngest enabled, during APM stats metric aggregation, only the resource attribute service.name would be used for service name, while the span attribute service.name attribute would be ignored.

This is believed to be a bug because, in Datadog backend, the service value in the span is still using span attribute service.name, while the service value in APM metrics uses the resource attribute service.name.

For these kind of spans, the APM metrics related UX would be broken because of the inconsistencies.

Steps to Reproduce

Send a span with attribute service.name value sname and resource attribute service.name value rname For example,

{
  "resourceSpans": [
    {
      "resource": {
        "attributes": [
          {
            "key": "service.name",
            "value": {
              "stringValue": "rname"
            }
          },
          {
            "key": "deployment.environment",
            "value": {
              "stringValue": "local"
            }
          }
        ]
      },
      "scopeSpans": [
        {
          "scope": {
            "name": "io.opentelemetry.http-url-connection"
          },
          "spans": [
            {
              "traceId": "71699b6fe85982c7c8995ea3d9c95df2",
              "spanId": "3c191d03fa8be065",
              "name": "spanitron",
              "kind": 3,
              "startTimeUnixNano": "1716776846359907000",
              "endTimeUnixNano": "1716776846360086000",
              "droppedAttributesCount": 0,
              "events": [],
              "droppedEventsCount": 0,
              "status": {
                "code": 1
              },
              "attributes": [
                {
                  "key": "service.name",
                  "value" : {
                    "stringValue": "sname"
                  }
                }
              ]
            }
          ]
        }
      ]
    }
  ]
 }

Expected Result

APM metrics with attr service:sname.

Actual Result

APM metrics with attr service:rname.

Collector version

v0.110.0

Environment information

Environment

OS: macOS Sonoma 14.6.1, arm64

OpenTelemetry Collector configuration

connectors:
  datadog/connector:
    traces:
      compute_stats_by_span_kind: true
      compute_top_level_by_span_kind: true
      peer_tags_aggregation: true
exporters:
  datadog:
    api:
      key: ${env:DD_API_KEY}
      site: datadoghq.com.
    host_metadata:
      enabled: false
    traces:
      compute_stats_by_span_kind: true
      compute_top_level_by_span_kind: true
      peer_tags_aggregation: true
  debug:
    verbosity: detailed
    sampling_initial: 5
    sampling_thereafter: 200
extensions:
  health_check:
    endpoint: ${env:MY_POD_IP}:13133
processors:
  batch: {}
  memory_limiter:
    check_interval: 5s
    limit_mib: 45
    spike_limit_percentage: 25
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: ${env:MY_POD_IP}:4317
      http:
        endpoint: ${env:MY_POD_IP}:4318
service:
  extensions:
  - health_check
  pipelines:
    metrics:
      receivers:
      - datadog/connector
      exporters:
      - datadog
    traces:
      exporters:
      - datadog/connector
      - datadog
      processors:
      - memory_limiter
      receivers:
      - otlp
  telemetry:
    metrics:
      address: ${env:MY_POD_IP}:8888

Log output

No response

Additional context

No response

[github-actions[bot]]

Pinging code owners:

• connector/datadog: @mx-psi @dineshg13 @ankitpatel96

  See Adding Labels via Comments if you do not have permissions to add labels yourself.

[songy23]

@tinyjacky if you can remove service.name from span attributes and only leave the one in resource attributes, that should solve the issue.

According to OTel spec https://opentelemetry.io/docs/specs/semconv/resource/#service service.name is a resource attribute rather than span attribute. Datadog semantic mapping doc suggests the same: https://docs.datadoghq.com/getting_started/tagging/unified_service_tagging/?tab=kubernetes#opentelemetry

When using OpenTelemetry, map the following resource attributes to their corresponding Datadog conventions

We plan to eventually remove the mappings that do not conform to OTel specs following some deprecation period, such as getting resource attributes from span attributes. /cc @IbraheemA @dineshg13