Open caseyg-ronin opened 6 months ago
Hi @caseyg-ronin
To help us investigate the issue, can you tell me more about the upstream service(s) talking to the application that raise this warn log.
The decision marker tag (t.dm
) looks wrong and I would like to know more about the service that emits it, especially which programming language is used and which version of the related Datadog client library (ie tracer).
I get the same thing using scala, cats-effect & redis
Hi @caseyg-ronin
To help us investigate the issue, can you tell me more about the upstream service(s) talking to the application that raise this warn log.
The decision marker tag (
t.dm
) looks wrong and I would like to know more about the service that emits it, especially which programming language is used and which version of the related Datadog client library (ie tracer).
What exactly are you looking for? The service is a fairly simple kotlin springboot API. No database. Talks to kafka and some other APIs. Most of the time, there's a corresponding API request that comes in, we process, write to kafka, then the error shows up. But it doesn't happen every time. And I don't know if that pattern is always consistent.
Here's the relevant datadog configuration (with some details redacted):
$ env | grep "DD_"
DD_DBM_PROPAGATION_MODE=full
DD_JMXFETCH_KAFKA_ENABLED=true
DD_JMXFETCH_ENABLED=true
DD_PROFILING_ENABLED=false
DD_LOGS_INJECTION=true
DD_ENV=dev
DD_VERSION=1.0.18
DD_RUNTIME_METRICS_ENABLED=true
DD_SERVICE_MAPPING=mysql:mysql-<service>
DD_DATA_STREAMS_ENABLED=true
DD_AGENT_HOST=<10.*.*.*>
DD_SERVICE=<service>
DD_TRACE_HEALTH_METRICS_STATSD_PORT=8125
DD_INTEGRATION_JDBC_DATASOURCE_ENABLED=true
DD_KAFKA_LEGACY_TRACING_ENABLED=false
I am looking for information about the service that calls your SpringBoot/Kotlin application through its API (the upstream service):
flowchart LR
A[Upstream service] --> |API| B[SpringBoot/Kotlin]
style A stroke:#f66,stroke-width:2px,stroke-dasharray: 5 5
B --> C[Kafka]
B --> |API| D[Downstream service]
It looks to be the one sending API requests with invalid headers to your SpringBoot app.
We have been experiencing the following WARN logs coming out of
dd-java-agent
that wraps our springboot service utilizingdd-trace-java
:Just upgraded both
dd-trace-java
anddd-java-agent
to1.30.0
and we're still seeing it. I think this is similar to https://github.com/DataDog/dd-trace-java/issues/5930, but maybe not exactly given the header value differences. Thought I'd submit it anyway to see if anyone has a fix.