search

open-telemetry / opentelemetry-lambda

Create your own Lambda Layer in each OTel language using this starter code. Add the Lambda Layer to your Lamdba Function to get tracing with OpenTelemetry.
https://opentelemetry.io
Apache License 2.0
277 stars 167 forks source link

Error when using AWS_LAMBDA_EXEC_WRAPPER '/opt/otel-proxy-handler' #270

Open msailes opened 2 years ago

msailes commented 2 years ago

I have a Lambda function with the following signature. These APIGateway classes represent the HTTP API version of API Gateway.

public class ApiGatewayGetAllProductRequestHandler implements RequestHandler<APIGatewayV2HTTPEvent, APIGatewayV2HTTPResponse> {

This causes an error with the OTEL Layer implementation using 'AWS_LAMBDA_EXEC_WRAPPER' = '/opt/otel-proxy-handler'

My Lambda function throws and error and doesn't complete and I get the following error message:

START RequestId: 0e4c8e0f-e0c9-49d0-83a9-43bdbb6a88ba Version: $LATEST
EXTENSION   Name: collector State: Ready    Events: [INVOKE,SHUTDOWN]
2022/08/04 12:53:29 [collector] Received event: {
    "eventType": "INVOKE",
    "deadlineMs": 1659617612128,
    "requestId": "0e4c8e0f-e0c9-49d0-83a9-43bdbb6a88ba",
    "invokedFunctionArn": "arn:aws:lambda:eu-west-1:123456789012:function:JVMPerfTestStack-GetAllProductFunction9216DFAB-EAi3pX6W38xp",
    "tracing": {
        "type": "X-Amzn-Trace-Id",
        "value": "Root=1-62ebc144-782d1b5979d1d6b165c4873f;Parent=1a8f548170e20f69;Sampled=1"
    }
}
2022/08/04 12:53:29 [collector] Waiting for event...
2022-08-04T12:53:29.309Z    INFO    loggingexporter/logging_exporter.go:56  MetricsExporter {"#metrics": 1}
Could not map API Gateway event body to requested parameter type: class com.amazonaws.services.lambda.runtime.events.APIGatewayV2HTTPEvent: java.lang.IllegalStateException
java.lang.IllegalStateException: Could not map API Gateway event body to requested parameter type: class com.amazonaws.services.lambda.runtime.events.APIGatewayV2HTTPEvent
    at io.opentelemetry.instrumentation.awslambdaevents.v2_2.TracingRequestApiGatewayWrapper.map(TracingRequestApiGatewayWrapper.java:41)
    at io.opentelemetry.instrumentation.awslambdaevents.v2_2.LambdaParameters.toArray(LambdaParameters.java:24)
    at io.opentelemetry.instrumentation.awslambdaevents.v2_2.TracingRequestWrapperBase.doHandleRequest(TracingRequestWrapperBase.java:63)
    at io.opentelemetry.instrumentation.awslambdaevents.v2_2.TracingRequestApiGatewayWrapper.doHandleRequest(TracingRequestApiGatewayWrapper.java:49)
    at io.opentelemetry.instrumentation.awslambdaevents.v2_2.TracingRequestApiGatewayWrapper.doHandleRequest(TracingRequestApiGatewayWrapper.java:21)
    at io.opentelemetry.instrumentation.awslambdacore.v1_0.TracingRequestHandler.handleRequest(TracingRequestHandler.java:79)
Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'eyJ0ZXN0IjoiYm9keSJ9': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
 at [Source: (String)"eyJ0ZXN0IjoiYm9keSJ9"; line: 1, column: 21]
    at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2337)
    at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:720)
    at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._reportInvalidToken(ReaderBasedJsonParser.java:2903)
    at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1949)
    at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:781)
    at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:4684)
    at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4586)
    at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3548)
    at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3516)
    at io.opentelemetry.instrumentation.awslambdaevents.v2_2.TracingRequestApiGatewayWrapper.map(TracingRequestApiGatewayWrapper.java:39)
    ... 5 more

END RequestId: 0e4c8e0f-e0c9-49d0-83a9-43bdbb6a88ba
REPORT RequestId: 0e4c8e0f-e0c9-49d0-83a9-43bdbb6a88ba  Duration: 257.58 ms Billed Duration: 258 ms Memory Size: 2048 MB    Max Memory Used: 211 MB Init Duration: 4067.70 ms   
XRAY TraceId: 1-62ebc144-782d1b5979d1d6b165c4873f   SegmentId: 1a8f548170e20f69 Sampled: true

The error comes from this line

https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/f236b2d4c9a0813b2dddf3d35928c79b21beb6d5/instrumentation/aws-lambda/aws-lambda-events-2.2/library/src/main/java/io/opentelemetry/instrumentation/awslambdaevents/v2_2/TracingRequestApiGatewayWrapper.java#L39

Where the code only expects 'APIGatewayProxyRequestEvent'. Other event objects will be used by customers and should be supported by the otel-proxy-handler.

willarmiros commented 2 years ago

Short term mitigation: Update https://aws-otel.github.io/docs/getting-started/lambda/lambda-java doc page to indicate only the 1 kind of APIGW event object is supported

Long term mitigation: actually fix the Lambda Java instrumentation code to support both types of objects.

github-actions[bot] commented 3 months ago

This issue was marked stale. It will be closed in 30 days without additional activity.

arun-annamalai commented 2 months ago

Any updates on a fix here?

tylerbenson commented 2 months ago

@arun-annamalai not yet. Any interest in submitting a PR fix?

serkan-ozal commented 2 months ago

@tylerbenson @msailes I think this issue should be fixed by this: https://github.com/open-telemetry/opentelemetry-java-instrumentation/pull/11868 With this PR, existing built-in serialization/deserialization mechanism is being used for the events. But to get the benefit of this PR, /opt/otel-handler must be used instead of /opt/otel-proxy-handler

serkan-ozal commented 1 month ago

@msailes I think, this issue should be fixed in the latest Java wrapper layer version and by using /opt/otel-handler instead of /opt/otel-proxy-handler. Can you verify that?

msailes commented 1 month ago

/opt/otel-handler handles both request types. Does that have the same functionality as /opt/otel-proxy-handler?

serkan-ozal commented 1 month ago

@msailes as far as I see, they are similar in terms of functionality. The only minor difference I can see is that when you return non APIGatewayProxyResponseEvent typed response, TracingRequestApiGatewayWrapper (used by /opt/otel-proxy-handler script) wraps it inside APIGatewayProxyResponseEvent, but TracingRequestWrapper (used by /opt/otel-handler script) doesn't.

msailes commented 1 month ago

@serkan-ozal the documentation at https://aws-otel.github.io/docs/getting-started/lambda/lambda-java#enable-auto-instrumentation-for-your-lambda-function seems to suggest that there is more functionality.

serkan-ozal commented 1 month ago

@msailes if you're talking about "HTTP context propagation" mentioned in the doc you shared, it is already supported by TracingRequestWrapper impl too. You can check the implementations here:

msailes commented 1 month ago

Fantastic, so it sounds like there is an opportunity to simplify the docs as well.

arun-annamalai commented 4 weeks ago

Here are a couple of the Lambda Java Event types related to APIGW. https://github.com/aws/aws-lambda-java-libs/blob/main/aws-lambda-java-events/README.md. I didnt fully follow the conversation of unification of otel lambda handlers. 

- APIGatewayCustomAuthorizerEvent
- APIGatewayProxyRequestEvent
- APIGatewayProxyResponseEvent
- APIGatewayV2CustomAuthorizerEvent
- APIGatewayV2HTTPEvent
- APIGatewayV2HTTPResponse
- APIGatewayV2WebSocketEvent
- APIGatewayV2WebSocketResponse

What handler is the correct one for APIGatewayV2HTTPEvent and APIGatewayV2HTTPResponse? Would that be /opt/otel-handler?