Running dotnet lambda publish-layer with --enable-package-optimization causes errors

jameswoodley commented 3 years ago

Description

If running dotnet lambda publish-layer with enable-package-optimization set to true I get errors that I don't get with it not set

Reproduction Steps

dotnet lambda publish-layer LuminescenceApiLambdaLayer --layer-type runtime-package-store --s3-bucket luminescence-api-lambda-layer --enable-package-optimization true

Logs

Amazon Lambda Tools for .NET Core applications (5.0.1) Project Home: https://github.com/aws/aws-extensions-for-dotnet-cli, https://github.com/aws/aws-lambda-dotnet

Enter Framework: (Target framework to compile, for example netcoreapp3.1.) net5.0 Warning: Package optimization has been enabled. Be sure to run this on an Amazon Linux environment or the optimization might not be compatbile with the Lambda runtime. Converted ASP.NET Core project file to temporary package manifest file. ... invoking 'dotnet store' for manifest /tmp/tmpGAAfJI.tmp into output directory /tmp/luminescenceapilambdalayer-637477173795722858/dotnetcore/store ... store: Microsoft (R) Build Engine version 16.8.3+39993bd9d for .NET ... store: Copyright (C) Microsoft Corporation. All rights reserved. ... store: Determining projects to restore... ... store: Restored /tmp/tmpGAAfJI.tmp (in 708 ms). ... store: Determining projects to restore... ... store: Determining projects to restore... ... store: Determining projects to restore... ... store: Determining projects to restore... ... store: Restored /tmp/tmpGAAfJI.tmp (in 16 ms). ... store: Restored /tmp/tmpGAAfJI.tmp (in 58 ms). ... store: Determining projects to restore... ... store: Determining projects to restore... ... store: Determining projects to restore... ... store: Determining projects to restore... ... store: Restored /tmp/tmpGAAfJI.tmp (in 277 ms). ... store: Determining projects to restore... ... store: Restored /tmp/tmpGAAfJI.tmp (in 276 ms). ... store: Restored /tmp/tmpGAAfJI.tmp (in 44 ms). ... store: Determining projects to restore... ... store: Determining projects to restore... ... store: Determining projects to restore... ... store: Restored /tmp/tmpGAAfJI.tmp (in 308 ms). ... store: Restored /tmp/tmpGAAfJI.tmp (in 357 ms). ... store: Restored /tmp/tmpGAAfJI.tmp (in 101 ms). ... store: Restored /tmp/tmpGAAfJI.tmp (in 272 ms). ... store: Restored /tmp/tmpGAAfJI.tmp (in 149 ms). ... store: Restored /tmp/tmpGAAfJI.tmp (in 152 ms). ... store: Restored /tmp/tmpGAAfJI.tmp (in 496 ms). ... store: Native image /tmp/bulwocn3.0tq/Amazon.Lambda.AspNetCoreServer_5.3.0/runtimopt/Microsoft.Extensions.Logging.Abstractions.dll generated successfully. ... store: Native image /tmp/bulwocn3.0tq/Amazon.Lambda.AspNetCoreServer_5.3.0/runtimopt/Microsoft.Extensions.Configuration.Abstractions.dll generated successfully. ... store: Native image /tmp/bulwocn3.0tq/Amazon.Lambda.AspNetCoreServer_5.3.0/runtimopt/Microsoft.Extensions.Configuration.Binder.dll generated successfully. ... store: Native image /tmp/bulwocn3.0tq/Amazon.Lambda.AspNetCoreServer_5.3.0/runtimopt/Amazon.Lambda.Logging.AspNetCore.dll generated successfully. ... store: Native image /tmp/bulwocn3.0tq/Amazon.Lambda.AspNetCoreServer_5.3.0/runtimopt/Amazon.Lambda.Core.dll generated successfully. ... store: Native image /tmp/bulwocn3.0tq/Amazon.Lambda.AspNetCoreServer_5.3.0/runtimopt/Amazon.Lambda.Serialization.SystemTextJson.dll generated successfully. ... store: Native image /tmp/bulwocn3.0tq/Amazon.Lambda.AspNetCoreServer_5.3.0/runtimopt/Microsoft.Extensions.Configuration.dll generated successfully. ... store: Successfully generated perfmap for native assembly '/tmp/bulwocn3.0tq/Amazon.Lambda.AspNetCoreServer_5.3.0/runtimopt/Microsoft.Extensions.Configuration.Abstractions.dll'. ... store: Successfully generated perfmap for native assembly '/tmp/bulwocn3.0tq/Amazon.Lambda.AspNetCoreServer_5.3.0/runtimopt/Microsoft.Extensions.Logging.Abstractions.dll'. ... store: Native image /tmp/bulwocn3.0tq/Amazon.Lambda.AspNetCoreServer_5.3.0/runtimopt/Microsoft.Extensions.Logging.dll generated successfully. ... store: Successfully generated perfmap for native assembly '/tmp/bulwocn3.0tq/Amazon.Lambda.AspNetCoreServer_5.3.0/runtimopt/Amazon.Lambda.Logging.AspNetCore.dll'. ... store: Successfully generated perfmap for native assembly '/tmp/bulwocn3.0tq/Amazon.Lambda.AspNetCoreServer_5.3.0/runtimopt/Amazon.Lambda.Core.dll'. ... store: Native image /tmp/bulwocn3.0tq/Amazon.Lambda.AspNetCoreServer_5.3.0/runtimopt/Amazon.Lambda.APIGatewayEvents.dll generated successfully. ... store: Successfully generated perfmap for native assembly '/tmp/bulwocn3.0tq/Amazon.Lambda.AspNetCoreServer_5.3.0/runtimopt/Microsoft.Extensions.Configuration.Binder.dll'. ... store: Successfully generated perfmap for native assembly '/tmp/bulwocn3.0tq/Amazon.Lambda.AspNetCoreServer_5.3.0/runtimopt/Amazon.Lambda.Serialization.SystemTextJson.dll'. ... store: Successfully generated perfmap for native assembly '/tmp/bulwocn3.0tq/Amazon.Lambda.AspNetCoreServer_5.3.0/runtimopt/Microsoft.Extensions.Configuration.dll'. ... store: Successfully generated perfmap for native assembly '/tmp/bulwocn3.0tq/Amazon.Lambda.AspNetCoreServer_5.3.0/runtimopt/Microsoft.Extensions.Logging.dll'. ... store: Successfully generated perfmap for native assembly '/tmp/bulwocn3.0tq/Amazon.Lambda.AspNetCoreServer_5.3.0/runtimopt/Amazon.Lambda.APIGatewayEvents.dll'. ... store: Error: Could not load file or assembly 'Microsoft.Extensions.DependencyInjection.Abstractions, Version=3.1.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The located assembly's manifest definition does not match the assembly reference. (0x80131040) ... store: Error compiling /home/wooderz/.nuget/packages/amazon.lambda.aspnetcoreserver/5.3.0/lib/netcoreapp3.1/Amazon.Lambda.AspNetCoreServer.dll: Could not find or load a specific file. (0x80131621) ... store: Error: compilation failed for "/home/wooderz/.nuget/packages/amazon.lambda.aspnetcoreserver/5.3.0/lib/netcoreapp3.1/Amazon.Lambda.AspNetCoreServer.dll" (0x80131621) ... store: /usr/share/dotnet/sdk/5.0.102/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.CrossGen.targets(148,5): error MSB3073: The command "/tmp/bulwocn3.0tq/Optimize/netcoreapp/crossgen -nologo -readytorun -in "/home/wooderz/.nuget/packages/amazon.lambda.aspnetcoreserver/5.3.0/lib/netcoreapp3.1/Amazon.Lambda.AspNetCoreServer.dll" -out "/tmp/bulwocn3.0tq/Amazon.Lambda.AspNetCoreServer_5.3.0/runtimopt/Amazon.Lambda.AspNetCoreServer.dll" -jitpath "/home/wooderz/.nuget/packages/microsoft.netcore.app.runtime.linux-x64/5.0.2/runtimes/linux-x64/native/libclrjit.so" -platform_assemblies_paths "/tmp/bulwocn3.0tq/Amazon.Lambda.AspNetCoreServer_5.3.0/runtimeref:/tmp/bulwocn3.0tq/Optimize/netcoreapp"" exited with code 33. [/tmp/tmpGAAfJI.tmp] ... store: Native image /tmp/bulwocn3.0tq/Amazon.Lambda.AspNetCoreServer_5.3.0/runtimopt/Amazon.Lambda.ApplicationLoadBalancerEvents.dll generated successfully. ... store: Native image /tmp/bulwocn3.0tq/Amazon.Lambda.AspNetCoreServer_5.3.0/runtimopt/Microsoft.Extensions.DependencyInjection.Abstractions.dll generated successfully. ... store: Successfully generated perfmap for native assembly '/tmp/bulwocn3.0tq/Amazon.Lambda.AspNetCoreServer_5.3.0/runtimopt/Amazon.Lambda.ApplicationLoadBalancerEvents.dll'. ... store: Successfully generated perfmap for native assembly '/tmp/bulwocn3.0tq/Amazon.Lambda.AspNetCoreServer_5.3.0/runtimopt/Microsoft.Extensions.DependencyInjection.Abstractions.dll'. ... store: Native image /tmp/bulwocn3.0tq/Amazon.Lambda.AspNetCoreServer_5.3.0/runtimopt/System.Runtime.CompilerServices.Unsafe.dll generated successfully. ... store: Native image /tmp/bulwocn3.0tq/Amazon.Lambda.AspNetCoreServer_5.3.0/runtimopt/Microsoft.Extensions.Primitives.dll generated successfully. ... store: Successfully generated perfmap for native assembly '/tmp/bulwocn3.0tq/Amazon.Lambda.AspNetCoreServer_5.3.0/runtimopt/System.Runtime.CompilerServices.Unsafe.dll'. ... store: Native image /tmp/bulwocn3.0tq/Amazon.Lambda.AspNetCoreServer_5.3.0/runtimopt/Microsoft.Extensions.Options.dll generated successfully. ... store: Successfully generated perfmap for native assembly '/tmp/bulwocn3.0tq/Amazon.Lambda.AspNetCoreServer_5.3.0/runtimopt/Microsoft.Extensions.Primitives.dll'. ... store: Successfully generated perfmap for native assembly '/tmp/bulwocn3.0tq/Amazon.Lambda.AspNetCoreServer_5.3.0/runtimopt/Microsoft.Extensions.Options.dll'. Error executing the 'dotnet store' command

Environment

WSL 2 Ubuntu 20.04 LTS Amazon Lambda Tools for .NET Core applications (5.0.1)

I'm sure it's something I've done, perhaps it's due to using dotnet 5? I'm attempting to create a lambda layer with my pre-jitted assemblies so I can then publish a docker container for a lambda and get below 2s cold starts.

This is a :bug: bug-report

ashishdhingra commented 3 years ago

Hi @jameswoodley,

Thanks for posting the issue. Could you please provide the sample code solution and the exact repro steps using that solution? Also, does it work when you target netcoreapp3.1 instead of net5.0?

I also see some errors in above output:

... store: Error: Could not load file or assembly 'Microsoft.Extensions.DependencyInjection.Abstractions, Version=3.1.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The located assembly's manifest definition does not match the assembly reference. (0x80131040)
... store: Error compiling /home/wooderz/.nuget/packages/amazon.lambda.aspnetcoreserver/5.3.0/lib/netcoreapp3.1/Amazon.Lambda.AspNetCoreServer.dll: Could not find or load a specific file. (0x80131621)
... store: Error: compilation failed for "/home/wooderz/.nuget/packages/amazon.lambda.aspnetcoreserver/5.3.0/lib/netcoreapp3.1/Amazon.Lambda.AspNetCoreServer.dll" (0x80131621)

Please advise if you are able to resolve these errors.

Thanks, Ashish

jameswoodley commented 3 years ago

Hi @ashishdhingra

As I mentioned this all works if not optimizing for Ready To Run. So I don't think these are code issues??

To me that looks like Amazon.Lambda.AspNetCoreServer has a dependency on Microsoft.Extensions.DependencyInjection.Abstractions that it is struggling to resolve?

jameswoodley commented 3 years ago

Hi @ashishdhingra

Just while I've been testing, I've found that the following package manifest will also fail, so maybe you could try and see if you can get anything working?

My environment if it helps is Ubuntu 20.04 LTS on WSL2

<Project Sdk="Microsoft.NET.Sdk">
    <PropertyGroup>
        <TargetFramework>netcoreapp3.1</TargetFramework>
        <GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
        <AWSProjectType>Lambda</AWSProjectType>
        <CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>
    </PropertyGroup>
    <ItemGroup>
        <PackageReference Include="Amazon.Lambda.AspNetCoreServer" Version="5.3.0" />
    </ItemGroup>
</Project>

ashishdhingra commented 3 years ago

Hi @ashishdhingra

Just while I've been testing, I've found that the following package manifest will also fail, so maybe you could try and see if you can get anything working?

My environment if it helps is Ubuntu 20.04 LTS on WSL2
<Project Sdk="Microsoft.NET.Sdk">

    <PropertyGroup>

        <TargetFramework>netcoreapp3.1</TargetFramework>

        <GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>

        <AWSProjectType>Lambda</AWSProjectType>

        <CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>

    </PropertyGroup>

    <ItemGroup>

        <PackageReference Include="Amazon.Lambda.AspNetCoreServer" Version="5.3.0" />

    </ItemGroup>

</Project>

Hi @jameswoodley,

Just curious, in package manifest, you have specified target framework as netcoreapp3.1, but while running the command from .NET CLI, you specify the framework as net5.0. Why these are different? Also for .NET 5.0, did you use the new server less template that uses docker file?

Thanks, Ashish

jameswoodley commented 3 years ago

I've changed to netcoreapp3.1 as per your suggestion. I do understand net5.0 isn't LTS or supported so thought netcoreapp3.1 just to try and get things working made sense

ashishdhingra commented 3 years ago

I've changed to netcoreapp3.1 as per your suggestion. I do understand net5.0 isn't LTS or supported so thought netcoreapp3.1 just to try and get things working made sense

@jameswoodley Please confirm if your issue is resolved.

jameswoodley commented 3 years ago

So far no.

However I've found some Microsoft nuget packages no longer support netcoreapp3.1, so I'm trying to go through and downgrade everything.

ashishdhingra commented 3 years ago

So far no.

However I've found some Microsoft nuget packages no longer support netcoreapp3.1, so I'm trying to go through and downgrade everything.

@jameswoodley My understanding was that you already had everything working for netcoreapp3.1 with all dependencies resolved and tried to upgrade to net5.0. But looks like you are trying to use packages that do not support netcoreapp3.1. Are you trying to develop from scratch with outdated packages?

jameswoodley commented 3 years ago

No I tried to develop from scratch with net5.0 packages, but that wouldn't work. So as per your suggestion I am now trying to downgrade everything to netcoreapp3.1

ashishdhingra commented 3 years ago

No I tried to develop from scratch with net5.0 packages, but that wouldn't work. So as per your suggestion I am now trying to downgrade everything to netcoreapp3.1

Please refer https://github.com/aws/aws-lambda-dotnet/tree/master/LambdaRuntimeDockerfiles for Lambda .NET 5 base image Readme. There is also a sample in the aws-lambda-dotnet/LambdaRuntimeDockerfiles/dotnet5/sample/ folder.

Also refer https://github.com/aws/aws-extensions-for-dotnet-cli/issues/155#issuecomment-761153859, there is a note on how to create .NET 5 lambda project from .NET CLI.

jameswoodley commented 3 years ago

That doesn't use Lambda Layers though does it? It's the dotnet lambda publish-layer command I'm having issues with

ashishdhingra commented 3 years ago

That doesn't use Lambda Layers though does it? It's the dotnet lambda publish-layer command I'm having issues with

Yes, you are correct. That's just an example on how develop using .NET 5 Lambda template which uses Docker and I wanted to point you to right direction if planning to use .NET 5.

Please comment here once your issue is resolved.

jameswoodley commented 3 years ago

I think this is the problem

Amazon.Lambda.AspNetCoreServer isn't targeting net5.0 yet, is there an upgrade planned?

ashishdhingra commented 3 years ago

I think this is the problem

Amazon.Lambda.AspNetCoreServer isn't targeting net5.0 yet, is there an upgrade planned?

Hi @jameswoodley,

As per previous links for .NET 5 Lambda support, it is supported as container image. Please refer developer blog article .NET 5 AWS Lambda Support with Container Images. For your case, after you install latest AWS Lambda Templates (latest version) using command dotnet new -i Amazon.Lambda.Templates, following templates are available (user dotnet new -all command for list):

Lambda Custom Runtime Function (.NET 5)
Lambda Empty Function (.NET 5 Container Image)
Lambda ASP.NET Core Web API (.NET 5 Container Image)
Lambda Empty Serverless (.NET 5 Container Image)

You could also refer the following articles if you want to use custom runtime for .NET 5:
.NET 5 on AWS
Exploring .NET 5 with the AWS Toolkit for Visual Studio

Hope this helps.

Thanks, Ashish

jameswoodley commented 3 years ago

I think I'm at the point where we decide this doesn't work in net5.0. I'm not asking for help on how to setup a lambda in net5, I've done that.. I'm asking about lambda layers, because the cold starts are far too slow as it stands

github-actions[bot] commented 3 years ago

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

carlin-q-scott commented 3 years ago

@jameswoodley Why did you close this issue? Did you resolve the dependency issues by downgrading everything to 3x MS packages?

jameswoodley commented 3 years ago

Yes, this wasn't an AWS issue, it was a Microsoft issue. The nuget packages didn't support .net5

aws / aws-extensions-for-dotnet-cli