[BUG] Tags from Agent are not applied to ingested OTLP data

[Okeanos]

Agent Environment

Using the Datadog helm-chart version 3.32.3

Datadog Agent version: 7.45.0

Edit: We have since updated to chart version 3.40.3 and agent version 7.48.1 with no noticeable difference.

Describe what happened:

We have the Datadog agent configured to allow ingestion of Open Telemetry (OTLP) data and also some generic tags that have to be applied to everything.

We use the opentelemetry-java-instrumentation Java Agent to send both trace data and (custom) metrics to the Datadog agent running in our Kubernetes cluster.

The traces and metrics themselves are available in Datadog as expected. Metrics, however, are missing all generic tags defined via the datadog.tags configuration at the agent level. For Traces, the traces themselves do not visibly show the datadogs.tags within the Trace details (APM Traces -> select any Trace -> Span Info / Span Tags view). However, traces can correctly be filtered at the generic APM Traces view level by inputting any of the datadog.tags key-value pairs. So the trace meta information at the Datadog-magic-level has successfully been enriched. This is invisible, though. Interestingly, the trace details (APM Traces -> select any Trace -> Span Info / Span Tags view) DOES contain the env:dev tag and only that.

Additionally, the OTLP resource attributes are correctly mapped & displayed as tags for both metrics and traces.

Describe what you expected:

We expect that the datadog.tags are consistently applied to all data reported to Datadog, including traces and (custom) metrics reported via the OTLP ingestion API. These tags should also be visible within the Datadog UI whenever tags of traces / metrics are viewed by a user.

The description of the YAML key says:

datadog:
  # datadog.tags -- List of static tags to attach to every metric, event and service check collected by this Agent.

  ## Learn more about tagging: https://docs.datadoghq.com/tagging/
  tags: []
  #   - "<KEY_1>:<VALUE_1>"
  #   - "<KEY_2>:<VALUE_2>"

Steps to reproduce the issue:

Configure the Datadog agent similar to this:

datadog:
  tags:
    # previously these were unquoted which obviously (in hindsight) doesn't work well for YAML.
    # this doesn't fix metrics, though
    - "region:westeurope"
    - "geo:foo"
    - "team:bar"
    - "env:dev"
  env:
    - name: DD_OTLP_GRPC_PORT
      value: 4317
    - name: DD_OTLP_HTTP_PORT
      value: 4318
    - name: DD_OTLP_CONFIG_METRICS_RESOURCE_ATTRIBUTES_AS_TAGS
      value: true
  otlp:
    receiver:
      protocols:
        grpc:
          enabled: true
        http:
          enabled: true

Then run a Java application with the opentelemetry-java-instrumentation (abbreviated to otel-agent.jar below) and have it send traces and custom metrics to the Datadog agent.

java \
  -javaagent:/otel-agent.jar \
  -Dotel.propagators=tracecontext,baggage,datadog \
  -Dotel.service.name=baz-bar \
  -Dotel.resource.attributes=service=baz-bar,team=foobar,key=bb,instrumentation.type=otel-agent,service.version=1.0.0 \
  -jar service.jar \

Then inspect the metrics via the "Metrics Summary" in Datadog and you will see that the (custom) metrics do not have the expected tags. Only the resource attributes will show up as tags (alongside some stuff like os.*, process.*, container.*, host.*) but NOT the datadog.tags-values.

Same with trace details viewed via the "APM Traces" view. Note: the APM Traces list can in fact be filtered correctly by the datadog.tags, but they aren't shown anywhere in the UI.

Additional environment details (Operating System, Cloud provider, etc):

Operating System: Linux Cloud Provider: Microsoft Azure Cloud Type: Kubernetes

[toaiduongdh]

We also face the same problem, any update on this?

[Okeanos]

@toaiduongdh we updated our environment to a more recent version of the Datadog Cluster agent but to the best of my knowledge that did not change anything and we still have to manually work around this issue (by providing ALL expected tags via e.g. the java agent's resource attributes parameter).

[onatm]

@toaiduongdh @Okeanos

I believe this is related to https://docs.datadoghq.com/opentelemetry/schema_semantics/hostname/?tab=datadogexporter#cloud-provider-specific-conventions

Edit: Commented on the wrong issue. Sorry for the confusion 😊

[Okeanos]

@onatm I am not sure I follow what you are trying to tell me. Nowhere in my initial report did I mention anything about host.name or that being a problem. I mentioned host.* tags being propagated correctly.

What I said was: that every DD tag shows up except for the ones defined in the datadog.tags list that has been attached to the Datadog Cluster agent. When having said Datadog Cluster agent collect and forward traces/metrics from the Java Open Telemetry instrumentation the values in datadog.tags do not appear in Datadog proper.

[ntrp]

Apparently this is how you can do it https://docs.datadoghq.com/opentelemetry/collector_exporter/hostname_tagging/?tab=host#custom-container-tags.

Basically you need to set an attribute: datadog.container.tag.<tag_name>=<tag_value> For example:

export OTEL_RESOURCE_ATTRIBUTES="datadog.container.tag.team=teamA"

Took me an entire day to figure it out, not really easy to find