search

DataDog / dd-trace-java

Datadog APM client for Java
https://docs.datadoghq.com/tracing/languages/java
Apache License 2.0
562 stars 282 forks source link

WARN `datadog.trace.agent.core.propagation.ptags.W3CPTagsCodec`: `Invalid datadog tags header value` #6725

Open caseyg-ronin opened 6 months ago

caseyg-ronin commented 6 months ago

We have been experiencing the following WARN logs coming out of dd-java-agent that wraps our springboot service utilizing dd-trace-java:

[dd.trace 2024-02-22 14:49:58:878 +0000] [http-nio-8080-exec-2] WARN datadog.trace.agent.core.propagation.ptags.W3CPTagsCodec - Invalid datadog tags header value: 'dd=t.tid:65d75f1600000000;t.dm:1;s:0' invalid tag value at 31 (Will not log warnings for 5 minutes)

Just upgraded both dd-trace-java and dd-java-agent to 1.30.0 and we're still seeing it. I think this is similar to https://github.com/DataDog/dd-trace-java/issues/5930, but maybe not exactly given the header value differences. Thought I'd submit it anyway to see if anyone has a fix.

PerfectSlayer commented 6 months ago

Hi @caseyg-ronin

To help us investigate the issue, can you tell me more about the upstream service(s) talking to the application that raise this warn log.

The decision marker tag (t.dm) looks wrong and I would like to know more about the service that emits it, especially which programming language is used and which version of the related Datadog client library (ie tracer).

Marcus-Rosti commented 6 months ago

I get the same thing using scala, cats-effect & redis

caseyg-ronin commented 6 months ago

Hi @caseyg-ronin

To help us investigate the issue, can you tell me more about the upstream service(s) talking to the application that raise this warn log.

The decision marker tag (t.dm) looks wrong and I would like to know more about the service that emits it, especially which programming language is used and which version of the related Datadog client library (ie tracer).

What exactly are you looking for? The service is a fairly simple kotlin springboot API. No database. Talks to kafka and some other APIs. Most of the time, there's a corresponding API request that comes in, we process, write to kafka, then the error shows up. But it doesn't happen every time. And I don't know if that pattern is always consistent.

Here's the relevant datadog configuration (with some details redacted):

$ env | grep "DD_"
DD_DBM_PROPAGATION_MODE=full
DD_JMXFETCH_KAFKA_ENABLED=true
DD_JMXFETCH_ENABLED=true
DD_PROFILING_ENABLED=false
DD_LOGS_INJECTION=true
DD_ENV=dev
DD_VERSION=1.0.18
DD_RUNTIME_METRICS_ENABLED=true
DD_SERVICE_MAPPING=mysql:mysql-<service>
DD_DATA_STREAMS_ENABLED=true
DD_AGENT_HOST=<10.*.*.*>
DD_SERVICE=<service>
DD_TRACE_HEALTH_METRICS_STATSD_PORT=8125
DD_INTEGRATION_JDBC_DATASOURCE_ENABLED=true
DD_KAFKA_LEGACY_TRACING_ENABLED=false
PerfectSlayer commented 6 months ago

I am looking for information about the service that calls your SpringBoot/Kotlin application through its API (the upstream service):

flowchart LR
    A[Upstream service] --> |API| B[SpringBoot/Kotlin]
    style A stroke:#f66,stroke-width:2px,stroke-dasharray: 5 5
    B --> C[Kafka]
    B --> |API| D[Downstream service]

It looks to be the one sending API requests with invalid headers to your SpringBoot app.