search

open-telemetry / opentelemetry-lambda

Create your own Lambda Layer in each OTel language using this starter code. Add the Lambda Layer to your Lamdba Function to get tracing with OpenTelemetry.
https://opentelemetry.io
Apache License 2.0
279 stars 167 forks source link

Same TraceID/Spans repeated Multiple Times in Java OTEL Agent #813

Open sanjyoth opened 1 year ago

sanjyoth commented 1 year ago

Describe the bug We are using AWS Java Lambda along with otel lambda ARN arn:aws:lambda:us-east-1:184161586896:layer:opentelemetry-javaagent-0_1_0:1 to auto instrument the application flow. Whenever lambda is triggered we are seeing same traceid and corresponding spans being sent multiple times to collector, in our case atleast 3 times which is an issue for us.

For example: Jul 19 17:21:10 ip-xx.ec2.internal otelcol[567034]: Trace ID : 2c2448453e8bb55f9aff1fc984d19c2b Jul 19 17:21:11 ip-xx.ec2.internal otelcol[567034]: Trace ID : 2c2448453e8bb55f9aff1fc984d19c2b Jul 19 17:21:11 ip-xx.ec2.internal otelcol[567034]: Trace ID : 2c2448453e8bb55f9aff1fc984d19c2b

Steps to reproduce Its a Spring Boot application having outbound calls to couple of other aws services like secrets, dynamodb

What did you expect to see? We were expecting to see 1 traceId and its corresponding spans only once

What did you see instead? We are seeing 3 calls being triggered every time lambda is triggered

What version of collector/language SDK version did you use? (arn:aws:lambda:us-east-1:184161586896:layer:opentelemetry-javaagent-0_1_0:1)

What language layer did you use? Java - Spring boot App

Additional context We saw x-ray was enabled in AWS Lambda, we did try to disable and run the same app but still the issue persists

ajaynagariya commented 1 year ago

Hi SIG/OTEL team

Please review this issue with priority and help in getting resolution as this issue causing a great impact to one of our big enterprise customer.

Your priority intervention is much appreciated!

ajaynagariya commented 1 year ago

@vasireddy99 , @codeboten , @svrnm

Need your attention and assistance on this

serkan-ozal commented 7 months ago

Hi @sanjyoth @ajaynagariya I am happy to help on this issue. How do you deploy your Spring boot app into AWS Lambda? Is it possible to share a sample project to reproduce this issue?

serkan-ozal commented 1 month ago

@ajaynagariya Are you saying that same span is reported multiple times, or there are different spans but you don't expect to see them?