ApiBootstrap .Net 7 AOT Error: Couldn't find valid bootstrap(s): [/var/task/bootstrap /opt/bootstrap] Runtime.InvalidEntrypoint

[tonywrfs]

Hi there,

I've just tried to get ApiBootstrap working from the MinimalApi solution. I deployed using the SAM template. It appears to have successfully deployed, but when I open the browser to the endpoint: https://myuniquegateway.execute-api.us-east-1.amazonaws.com/ it returns {"message":"Internal Server Error"}.

CloudWatch has the following error: Error: Couldn't find valid bootstrap(s): [/var/task/bootstrap /opt/bootstrap] Runtime.InvalidEntrypoint

I am using Visual Studio 2022 Professional, and I opened a terminal window. Target framework is .NET 7.0, Target OS is (None). Assembly name is bootstrap. The Platform target is x64. Optimize code is set to Release. I did a complete Rebuild of the Solution.

I executed the following commands to deploy the Lambda functions, and they executed successfully:

aws cloudformation package --output-template-file packaged.yaml --template-file template.yaml --s3-bucket my-deploymentss3bucket-xyzzymybucket --s3-prefix my-minimalapi --region us-east-1 
aws cloudformation deploy --template-file packaged.yaml --stack-name my-DotNet7MinimalApiStackv2 --region us-east-1 --capabilities CAPABILITY_IAM

I got the invoke url endpoint from the API gateway, of the form shown at the top (obviously not going to put my real url!)

How can I fix this?

[Beau-Gosse-dev]

Hi @tonywrfs, the best way to troubleshoot this is to go into the AWS console and export the deployment package. Or it looks like you're uploading the package to S3, so you can download it from there.

Then when you unzip it, there should be a file called bootstrap right in the root of the zip file. If there are sub-directories which contain the bootstrap file, or if the assembly wasn't called 'bootstrap', then Lambda won't be able to find the correct file to start inside the package at runtime.

You can set the assembly name in the project file like this, or rename it manually.

If there are sub-directories, you'll want to make sure you're zipping it in a way that the bootstrap file is at the root. Please let me know if that's the case, as we may need to update the aws cloudformation package command.

If that doesn't point you in the right direction, you should also make sure everything is x64 (not ARM). We hope to support ARM in the future, but it's currently blocked by a glibc version issue.

Please let me know what you see in the deployment package.

[tonywrfs]

Hi Beau, thanks for responding. I have attached the tree structure generated by the project in VS2022. I notice in the package folder e66612cfaea0082ec8cb160329ae7eb9\bin\Release\net7.0\win-x64 there is a bootstrap.exe and a bootstrap.dll. I suspect the extensions aren't needed, but is this the folder I need to reference? Also, I was thinking perhaps using a Post-build event to copy the file to the root, assuming there's not a better way?

mytree.txt

[Beau-Gosse-dev]

@tonywrfs I wouldn't expect so many library dlls to be included in the output. Also I'd expect the output to be under some folder called native by default. Such as bin\Debug\net7.0\win-x64\native

Is the bootstrap exe file in the MBs or KBs for size? If it's in the KBs, then it's definitely not a self-contained executable and probably isn't being published as native aot. Can you share your project file? Does the project file include <PublishAot>true</PublishAot> ?

[tonywrfs]

Hi Beau,

I've attached the csproj file. bootstrap.exe in the bin/release/net7.0/win-x64 folder is 151KB. bootstrap.dll is 14KB.

PublishAot is definitely set to true in the project file.

ApiBootstrapcsproj.txt

[tonywrfs]

Oh, and there's no "native" folder. fyi, I am running on Windows 11.

[Beau-Gosse-dev]

@tonywrfs I haven't used the aws cloudformation package command with .NET before, but I'm starting to suspect that it won't automatically publish the app for you. You mentioned using the SAM template, but it seems like you're not using sam deploy or sam deploy --guided to actually deploy the app. You also said you did a complete rebuild of the solution, but unless something is calling dotnet publish, you won't get the native binaries. sam deploy should know how to call dotnet publish and locate the output folder. I'm not sure aws cloudformation package will be know how to publish the .NET project on it's own.

[tonywrfs]

Alright, it turned out I needed to have Desktop Development for C++ installed in Visual Studio. I generally only install what I need on my machine, and I didn't need that, until now.

I then published the application to a local folder. It has four files: aspnetcorev2_inprocess.dll (371 KB) bootstrap.exe (~34.5MB) bootstrap.pdb (~141MB) Shared.pdb (70KB)

I then logged in to aws from the commandline, and executed sam deploy --guided It successfully executes a bunch of commands, but then I get this, and it rolls back:

CREATE_FAILED AWS::Lambda::Function ApiFunction Resource handler returned message: "Unzipped size must be smaller than 262144000 bytes (Service: Lambda, Status Code: 400, Request ID: f5e28195-ed3b-4af0-a094-8d3d5d2852ff)" (RequestToken: d8053aca-7037-9d1d-9788-2a8451f86568, HandlerErrorCode: InvalidRequest)

I then zipped up the contents of the native folder and deployed that instead, and it also failed with the same error {"message":"Internal Server Error"}.

Next, I started from scratch with a new .net 7 AOT application in Visual Studio 2022. I built it, and then did a right-click Publish to AWS Lambda, and published the default function to AWS. With the Desktop Development for C++ installed, this succeeded. I was able to test the function locally, and it worked.

I will go through and apply all the changes to a new .net 7 aot lamda function and see how I go.

Edit: By the way, the zip file that was created by Visual Studio Publish to AWS Lambda only contains a single file: bootstrap, which is just under 5MB.

[Beau-Gosse-dev]

Yes, native binaries can be very large since they contain the all of the necessary runtime components too. I see .pdb files in your output, we actually remove the debugging symbols to keep the size down. You can see that code here. To remove the debugging symbols on Linux, they need to be split out into their own file. You can do that with the StripSymbols project property as shown in our blueprint.

However, you have .pdb instead of .dbg indicating you're building on Windows. That can work for some troubleshooting, but Lambda runs on Linux and will need to be build on Linux. Both SAM CLI and Amazon.Lambda.Tools should automatically build on a Docker container if not already on Amazon Linux 2. You'll need Docker installed and running. But since you said the deployment worked from Visual Studio, you should already be setup there.

If the binary size is still too large, you can consider deploying a container image to Lambda, or uploading to S3. Also, more aggressive trimming should cut down on binary size. Trimming can be configured with these properties, and also by remove lines out of the rd.xml which tell the trimmer not to trim assemblies, if you have that file still. (rd.xml is no longer recommended).

Let me know if you have any other issues.