Closed jmichalek132 closed 1 year ago
Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself.
This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers
. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.
Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself.
This was fixed by #17399
Component(s)
exporter/datadog
What happened?
Description
A certain amount of log lines have datadog tags containing metadata of the wrong k8s pod when batching is enabled. The problem lies on this line: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/b2d2792014000486d6499f6bbe32e5a4ce34ec05/exporter/datadogexporter/internal/logs/sender.go#L73
We copy the attributes only from the first item in the payload and set them as datadog tags for all the log lines in the payload. However, when we enable batching the payload can contain logs from multiple different pods. When this happens all the share the same tags even though they are from different pods making the tags incorrect.
By setting a breakpoint inside this function we can inspect the payload sent and verify the issue. https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/b2d2792014000486d6499f6bbe32e5a4ce34ec05/exporter/datadogexporter/internal/logs/sender.go#L79 Snippet containing the function
This screenshot shows the issue when inspecting the payload sent using a debugger. We have body containing logs from different pods and different
ddtags
and theo
which contains only one set ofddtags
. Theo
one seems to be preferred.Steps to Reproduce
We ran otel collector as daemonset to collect logs traces and metrics and use otlp to send it to another otel collector ran as deployment for additional processing.
Expected Result
Properties such as pod name in tags match the pod name in attributes.
Actual Result
Example log line exported from datadog ui after ingestion: You can see for example that the pod name in tags doesn't match the pod name in attributes. The pod name in attributes is the correct one.
Collector version
v0.67.0
Environment information
Environment
OS: (e.g., "Ubuntu 20.04") Compiler(if manually compiled): (e.g., "go 14.2")
k8s version: Server Version: version.Info{Major:"1", Minor:"23+", GitVersion:"v1.23.13-eks-fb459a0", GitCommit:"55bd5d5cb7d32bc35e4e050f536181196fb8c6f7", GitTreeState:"clean", BuildDate:"2022-10-24T20:35:40Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"} Helm chart used
OpenTelemetry Collector configuration
Log output
No response
Additional context
I can provide more details if necessary.