Closed dhanvsagar closed 1 year ago
@dhanvsagar thanks for reporting this! I've seen this happen before and I believe it is an issue with the Go auto-instrumentation agent and http. I see you've opened an issue for this in the Go Instrumentation repository as well so I will close this for now. Please ping me if something changes and the operator becomes the suspect.
Describe the bug
Hi, I am have a demo app (https://github.com/GoogleCloudPlatform/microservices-demo) deployed in a k8s cluster. The services are in Java, NodeJS, .Net and Go.
I have the open telemetry operator installed in my cluster with featureGates: "operator.autoinstrumentation.go" to enable support for Go.
The app YAML files contains annotations as follows
My configured trace flow is as follows
[App + Auto Instrumentation] ---> OTel collector deployment --> Tempo --> Grafana
collector manifest as follows
Instrumentation.yaml
Environment
opentelemetry-operator-0.31.0
App Version 0.78.0
To Reproduce
Steps to reproduce the behavior:
Expected behavior
Traces must be flowing correctly to the collector and to tempo.
Actual behavior
I can see the auto-instrumentation sidecar gets created and
Update
Further looking into this, found that when I access the service, the spanID and traceID are generated as zeros/empty.
The OTel collector is unable to export the traces to tempo, and throws the error
{"kind": "exporter", "data_type": "traces", "name": "otlp", "error": "Permanent error: rpc error: code = InvalidArgument desc = trace ids must be 128 bit", "dropped_items": 6}
Error log from the Otel Collector
Error log from Tempo
Please note that this error is only when I enable auto-instrumentation for the Go services and I don't see this error for NodeJS, Python and Java services.
Could you please help to resolve the issue.