Open keithpeck opened 10 months ago
I'm not sure where the data is being reported to, but the delay might be caused by the synchronous flush.
Please try the new decouple
processor included in the latest release of the community collector to see if that helps.
For more details see the links below: https://github.com/open-telemetry/opentelemetry-lambda/pull/959 https://github.com/open-telemetry/opentelemetry-lambda/pull/1046 https://github.com/open-telemetry/opentelemetry-lambda/blob/main/collector/README.md#improving-lambda-responses-times
Hi @keithpeck, I suspect that coldstart delay might be caused by collector. Not sure how it is important in terms of delay, but as far as I see, collector gets config from S3 initially, so with 256MB memory, this might add 1-2 seconds additional delay.
Any leads on this? I have opened this ticket where a local config seems to be good and fast until Everything is ready. Begin running and processing data AND then there is a wait of about 1-2 seconds (BLACBOX). After that the lambda Lifecycle API shows EXTENSION Name: collector State: Ready Events: [INVOKE, SHUTDOWN]
Does anyone know where is this delay coming from? @serkan-ozal @tylerbenson
Describe the bug
When aws-otel-python layer is used, Lambda timeouts will cause high initialization times of the next Lambda call.
When our database is under high load, it can cause our Lambdas to timeout. In these high load situations we are also seeing very high Lambda initialization times of 19 seconds! Importantly, this initialization time is charged for and is removed from our remaining runtime - with a 30 second Lambda timeout,
context.get_remaining_time_in_millis()
is reporting only 11 seconds when handler is called.I have tested that these high initialization times are only present when the aws-otel-python layer is deployed and configured. I don't have experience with other extensions so am not able to confirm if this is specifically an open tel layer issue or something inherent to extensions
Steps to reproduce
This can be reproduced with a
time.sleep
call in the Lambda handler, and invoking the Lambda with some load so that it is reused.Normal initialization times of this example function are 1.2 seconds, with the outliers being 9 seconds.
And CW showing that function with 15 second timeout only had 6 seconds to run
CloudFormation and Python code is shared below which can be used to reproduce. Warnings:
You should expect to see printed results like this:
What version of collector/language SDK version did you use? arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-python-amd64-ver-1-20-0:2
What language layer did you use? Python 11
Additional context We attempted to mitigate this issue by setting lower statement timeouts on our DB calls, but under high DB load this timeout does not fire accurately.