Comments to "Cold Starts in AWS Lambda"

[mikhailshilkov]

Add your comment to Cold Starts in AWS Lambda. The comments will be displayed directly on the page. I may edit or merge your comments in a non-significant way to make sure they are displayed properly on the website.

[hrchu]

awsome!

[basezen]

This is wonderfully thorough and pinpoints my issue. Please keep up to date with workarounds!

[phillip-haydon]

How are you creating your C# Lambda? The cold starts for C# make no sense. The only time I've been able to get 5s+ start time on C# was including chromium in my package for puppeteer bloating the package to 42mb.

[mikhailshilkov]

Yes, I'm also suspicious about my C# results at this point. The code for my function is here. I just build it with dotnet and deploy the publish folder. The rest is the same as for all other languages. Any ideas why I might be off?

[phillip-haydon]

Hey,

So I took your code, I changed nothing, and ran:

dotnet lambda package -o banana.zip -c Release

Then I uploaded it and executed it, then saved it to make it cold-start again and executed, this was my timings with the lambda configured to default of 512mb of memory.

• 764.53 ms
• 755.16 ms
• 773.27 ms

At 128mb of memory

• 3291.64 ms
• 3306.33 ms
• 3305.16 ms

The lambda's I have are all configured for 512 for .net and 256 for node. I'm not sure what your settings are. But even at 128mb for .net I think your timings should be lower than what's graphed.

The build:

G:\github\cloudbench\aws\http\csnoop [master ≡]> dotnet lambda package -o banana.zip -c Release
Amazon Lambda Tools for .NET Core applications (3.3.1)
Project Home: https://github.com/aws/aws-extensions-for-dotnet-cli, https://github.com/aws/aws-lambda-dotnet

Executing publish command
... invoking 'dotnet publish', working folder 'G:\github\cloudbench\aws\http\csnoop\bin\Release\netcoreapp2.1\publish' ... Disabling compilation context to reduce package size. If compilation context is needed pass in the "/p:PreserveCompilationContext=false" switch.
... publish: Microsoft (R) Build Engine version 15.9.20+g88f5fadfbe for .NET Core
... publish: Copyright (C) Microsoft Corporation. All rights reserved.
... publish:   Restoring packages for G:\github\cloudbench\aws\http\csnoop\app.csproj...
... publish:   Restoring packages for G:\github\cloudbench\aws\http\csnoop\app.csproj...
... publish:   Restore completed in 1.3 sec for G:\github\cloudbench\aws\http\csnoop\app.csproj.
... publish:   Installing Amazon.Lambda.Serialization.Json 1.1.0.
... publish:   Generating MSBuild file G:\github\cloudbench\aws\http\csnoop\obj\app.csproj.nuget.g.props.
... publish:   Generating MSBuild file G:\github\cloudbench\aws\http\csnoop\obj\app.csproj.nuget.g.targets.
... publish:   Restore completed in 2.04 sec for G:\github\cloudbench\aws\http\csnoop\app.csproj.
... publish: Functions.cs(32,52): warning CS1998: This async method lacks 'await' operators and will run synchronously. Consider using the 'await' operator to await non-blocking API calls, or 'await Task.Run(...)' to do CPU-bound work on a background thread. [G:\github\cloudbench\aws\http\csnoop\app.csproj]
... publish:   app -> G:\github\cloudbench\aws\http\csnoop\bin\Release\netcoreapp2.1\rhel.7.2-x64\app.dll
... publish:   app -> G:\github\cloudbench\aws\http\csnoop\bin\Release\netcoreapp2.1\publish\
Zipping publish folder G:\github\cloudbench\aws\http\csnoop\bin\Release\netcoreapp2.1\publish to G:\github\cloudbench\aws\http\csnoop\banana.zip
... zipping: Amazon.Lambda.APIGatewayEvents.dll
... zipping: Amazon.Lambda.Core.dll
... zipping: Amazon.Lambda.Serialization.Json.dll
... zipping: app.deps.json
... zipping: app.dll
... zipping: app.pdb
... zipping: app.runtimeconfig.json
... zipping: Newtonsoft.Json.dll
Created publish archive (G:\github\cloudbench\aws\http\csnoop\banana.zip).
Lambda project successfully packaged: G:\github\cloudbench\aws\http\csnoop\banana.zip

[mikhailshilkov]

So, my mistake was jamming the data for all memory sizes from 128 to 2048 MB into one value. This works just fine for other runtimes, but not .NET! I changed the article to use the 2 GB value for the main language chart, and then added a chart of how cold start time of C# functions depends on the instance size. Thank you for pushing me in the right direction!

[phillip-haydon]

Awesome. Yeah c# is weird. It’s execution time is insanely fast but its startup is slow. What the graph shows is closer to what I experience now.

Will be interesting to see the difference with .net 3 but I don’t think we will see it until the LTS package is ready.

Thanks for taking the time to update your results.

[AnatoliiStepaniuk]

First plot "Typical cold start durations per language" Could you please specify the memory size of functions for this plot?

Same for the last plot "Cold start durations of the same Lambda with and without VPC access" What is the languages and function memory size?

[mikhailshilkov]

First plot "Typical cold start durations per language" Could you please specify the memory size of functions for this plot?

They are combined charts for all memory sizes. As "Does Instance Size Matter?" section show, HelloWorld-type functions have the same cold start distributions for any memory size, with .NET being one important deviation.

"Cold start durations of the same Lambda with and without VPC access" What is the languages and function memory size?

It's JavaScript, I'll specify this during the next refresh. Again, all memory sizes combined.

[klimchuk]

To avoid cold start penalty is it necessary for lambda to be outside of VPC?

[a-stepaniuk]

To avoid cold start penalty is it necessary for lambda to be outside of VPC?

@klimchuk According to AWS announcement, Lambda Coldstart should not be affected by VPC https://aws.amazon.com/blogs/compute/announcing-improved-vpc-networking-for-aws-lambda-functions/

[dsaini17]

Good one mate!

[andrewaggb]

I haven't written/tried a lambda yet, but the template for the custom runtime c# lambda includes this in the readme. Have any of you tried it?

Improve Cold Start

.NET Core 3.0 has a new feature called ReadyToRun. When you compile your .NET Core 3.0 application you can enable ReadyToRun to prejit the .NET assemblies. This saves the .NET Core runtime from doing a lot of work during startup converting the assemblies to a native format. ReadyToRun must be used on the same platform as the platform that will run the .NET application. In Lambda's case that means you have to build the Lambda package bundle in a Linux environment. To enable ReadyToRun edit the aws-lambda-tools-defaults.json file to add /p:PublishReadyToRun=true to the msbuild-parameters parameter.

[ar-arkhipov]

Thank you for the article, it's great! I would like to use few pictures (graphs) from the article in my presentation about AWS Lambda. I will, of course, mention your site as the source. Are you OK with that? Thanks.

[mikhailshilkov]

Sure, go for it, it's all open!

[unaiur]

Can you include numbers about new Java 11 support? Does it improve cold start time?

[mikhailshilkov]

I've updated the article - the latest tests were run on Java 11 runtime

[unaiur]

Great! Have you watched this awesome AWS re:invent video https://youtu.be/ddg1u5HLwg8 about optimizing Java cold starts? My lambda cold start time (which writes to dynamoDB and sends some SNS and SQS message) was reduced from 12 to 2 seconds following that advise!

El El vie, 1 may 2020 a las 21:53, Mikhail Shilkov notifications@github.com escribió:

I've updated the article - the latest tests were run on Java 11 runtime

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/mikhailshilkov/mikhailio-hugo/issues/2#issuecomment-622538377, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABM4WB4FXKUARTCU5DQS5GTRPMSCJANCNFSM4GZHDWFA .

[bersena911]

What if I take my python packages to layer, will it affect cold start?

[mikhailshilkov]

I haven't run experiments involving layers, but studying other sources suggests that layers do not improve the cold starts. But again, I have no data yet.

[jschulenklopper]

Great analysis. Recently, I ran another experiment - write-up at https://xebia.com/blog/til-that-aws-lambda-terminates-instances-preemptively/ - to test the average lifetime of active AWS Lambda instances. Bottom line is that Lambda is preemptively terminating active, running instances after about two hours, thus triggering a 'forced' cold-start for the next invocation. It seems you cannot let a Lambda instance run for longer than two hours. (Not that you should expect or plan for instances to run forever, but it was a surprise to me that Lambda is actively terminating instances.)

BTW, the initialization time for a cold-start isn't charged. So, you get the 'penalty' of an incidental higher latency because of a cold-start, but the duration of cold-starts aren't billed to your account.

[RileyMShea]

From the article, it seems once a lambda zipped deployment size is > ~10MB, it's better to use a docker container, since the image size doesn't influence cold start duration. Is that accurate?

[mikhailshilkov]

That's a great question... I should test this explicitly. My Docker images were filled with data files (random text). Having extra npm modules may contribute to other stages of a cold start, like loading the modules in memory.

[ahtokca]

Mikhail, thanks a lot

[BellBat]

This is a great article and very useful. It would be great if it could be updated with the latest .net runtime support on 3.1 w/ new serializer / readytorun https://aws.amazon.com/blogs/compute/announcing-aws-lambda-supports-for-net-core-3-1/

[rcollette]

However, for Lambda functions that do real world work, the testing shows a significant cold start improvement in .NET 6. For example, a .NET 6 Lambda function that uses the AWS .NET SDK to retrieve an item from DynamoDB showed a 25% performance improvement.

https://aws.amazon.com/blogs/compute/introducing-the-net-6-runtime-for-aws-lambda/

Hello world's are no longer sufficient for measuring cold start performance. Even Node has to JIT code. On a larger code base, with ReadyToRun, .NET might cold start better than Node.

[Co2Link]

I think the process of unarchiving in the zipped deployment is the reason why the cold start duration increases as the deployment size increases. As for container deployment, it doesn't inolve with unarchiving, so code start duration stays the same.

[nicornk]

Regarding container cold start times - AWS is lazy-loading those images: https://aws.amazon.com/about-aws/whats-new/2022/09/introducing-seekable-oci-lazy-loading-container-images/ That's probably the reason why you don't see big differences in the "cold start durations per docker image size extra" chart.

[OndroMih]

From the article, it seems once a lambda zipped deployment size is > ~10MB, it's better to use a docker container, since the image size doesn't influence cold start duration. Is that accurate?

What actually matters is the size of the application itself. In other words, how many files needs to be processes (compiled, interpreted) before the application can serve requests. If you wrap the same zipped lambda into a docker container, it will still start slow, probably even slower with the docker container overhead, which lacks runtime-specific improvements in AWS Lambda.

If you just add a 5GB of files into a docker container, and then the files aren't touched by the application, that won't impact the startup time. But if you add 5GB of libraries, which will be loaded by the application, that would increase the startup time dramatically. And it doesn't matter if you add it to the zipped deployment or into a docker container.