search

aws / aws-lambda-dotnet

Libraries, samples and tools to help .NET Core developers develop AWS Lambda functions.
Apache License 2.0
1.58k stars 479 forks source link

Duplicate response body when running on Lambda and manipulating response body stream in middleware #1185

Open madmox opened 2 years ago

madmox commented 2 years ago

Describe the bug

I have a ASP.NET middleware which temporarily replaces the HTTP response body stream with a memory stream so that it can perform work involving seek operations on the response stream before the response is sent to the client. The code works properly when running locally using Kestrel or the integration tests server, but when running on AWS, all response bodies are duplicated (i.e. {"foo":"bar"} becomes {"foo":"bar"}{"foo":"bar"}).

Expected Behavior

The response should be a valid JSON result on all hosting environments (e.g. {"foo":"bar"}).

Current Behavior

The response is invalid when running on AWS Lambda only (duplicated JSON, e.g. {"foo":"bar"}{"foo":"bar"}).

Reproduction Steps

Simplified middleware code looks like this:

public async Task InvokeAsync(HttpContext context)
{
    Stream originalResponseBodyStream = context.Response.Body;
    using var buffer = new MemoryStream();
    context.Response.Body = buffer;

    try
    {
        await this._next(context);
        // Do work involving seek on body stream
    }
    finally
    {
        context.Response.Body.Seek(0, SeekOrigin.Begin);
        await context.Response.Body.CopyToAsync(originalResponseBodyStream);
        context.Response.Body = originalResponseBodyStream;
    }
}

Possible Solution

No response

Additional Information/Context

My ASP.NET application is using the Minimal API hosting pattern. The AWS Lambda hosting mode is set to LambdaEventSource.RestApi.

AWS .NET SDK and/or Package version used

Targeted .NET Platform

.NET 6

Operating System and version

AmazonLinux

madmox commented 2 years ago

Sample repo to reproduce the issue.

ashishdhingra commented 2 years ago

Hi @madmox,

Good afternoon.

Thanks for sharing the sample code. Unfortunately, I'm unable to reproduce the issue. For reproduction,

Build succeeded.

D:\Lambda_Issue1185\awslambdadotnetissue1185\src\AWSLambdaTest\IndexController.cs(13,42): warning CS1998: This async method lacks 'await' operators and will run synchronously. Consider using the 'await' operator to await non-blocking API calls, or 'await Task.Run(...)' to do CPU-bound work on a background thread. [D:\Lambda_Issue1185\awslambdadotnetissue1185\src\AWSLambdaTest\AWSLambdaTest.csproj] 1 Warning(s) 0 Error(s)

Time Elapsed 00:00:06.91 Amazon Lambda Tools for .NET Core applications (5.3.0) Project Home: https://github.com/aws/aws-extensions-for-dotnet-cli, https://github.com/aws/aws-lambda-dotnet

Executing publish command ... invoking 'dotnet publish', working folder 'D:\Lambda_Issue1185\awslambdadotnetissue1185\src\AWSLambdaTest\bin\Release\net6.0\publish' ... dotnet publish "D:\Lambda_Issue1185\awslambdadotnetissue1185\src\AWSLambdaTest" --output "D:\Lambda_Issue1185\awslambdadotnetissue1185\src\AWSLambdaTest\bin\Release\net6.0\publish" --configuration "Release" --framework "net6.0" /p:GenerateRuntimeConfigurationFiles=true --runtime linux-x64 --self-contained false ... publish: Microsoft (R) Build Engine version 17.1.1+a02f73656 for .NET ... publish: Copyright (C) Microsoft Corporation. All rights reserved. ... publish: Determining projects to restore... ... publish: Restored D:\Lambda_Issue1185\awslambdadotnetissue1185\src\AWSLambdaTest\AWSLambdaTest.csproj (in 493 ms). ... publish: D:\Lambda_Issue1185\awslambdadotnetissue1185\src\AWSLambdaTest\IndexController.cs(13,42): warning CS1998: This async method lacks 'await' operators and will run synchronously. Consider using the 'await' operator to await non-blocking API calls, or 'await Task.Run(...)' to do CPU-bound work on a background thread. [D:\Lambda_Issue1185\awslambdadotnetissue1185\src\AWSLambdaTest\AWSLambdaTest.csproj] ... publish: AWSLambdaTest -> D:\Lambda_Issue1185\awslambdadotnetissue1185\src\AWSLambdaTest\bin\Release\net6.0\linux-x64\AWSLambdaTest.dll ... publish: AWSLambdaTest -> D:\Lambda_Issue1185\awslambdadotnetissue1185\src\AWSLambdaTest\bin\Release\net6.0\publish\ Zipping publish folder D:\Lambda_Issue1185\awslambdadotnetissue1185\src\AWSLambdaTest\bin\Release\net6.0\publish to D:\Lambda_Issue1185\awslambdadotnetissue1185\artifacts\AWSLambdaTest.zip Creating directory D:\Lambda_Issue1185\awslambdadotnetissue1185\artifacts ... zipping: Amazon.Lambda.APIGatewayEvents.dll ... zipping: Amazon.Lambda.ApplicationLoadBalancerEvents.dll ... zipping: Amazon.Lambda.AspNetCoreServer.dll ... zipping: Amazon.Lambda.AspNetCoreServer.Hosting.dll ... zipping: Amazon.Lambda.Core.dll ... zipping: Amazon.Lambda.Logging.AspNetCore.dll ... zipping: Amazon.Lambda.RuntimeSupport.dll ... zipping: Amazon.Lambda.Serialization.SystemTextJson.dll ... zipping: appsettings.json ... zipping: AWSLambdaTest ... zipping: AWSLambdaTest.deps.json ... zipping: AWSLambdaTest.dll ... zipping: AWSLambdaTest.pdb ... zipping: AWSLambdaTest.runtimeconfig.json Created publish archive (D:\Lambda_Issue1185\awslambdadotnetissue1185\artifacts\AWSLambdaTest.zip). Lambda project successfully packaged: D:\Lambda_Issue1185\awslambdadotnetissue1185\artifacts\AWSLambdaTest.zip Amazon Lambda Tools for .NET Core applications (5.3.0) Project Home: https://github.com/aws/aws-extensions-for-dotnet-cli, https://github.com/aws/aws-lambda-dotnet

Processing CloudFormation resource ServerlessApi Processing CloudFormation resource AspNetCoreFunction Initiate packaging of . for resource AspNetCoreFunction Uploading to S3. (Bucket: testbucket-issue1880 Key: AWSLambdaTest-637886578304268713.zip) ... Progress: 45% ... Progress: 91% ... Progress: 100% Uploading to S3. (Bucket: testbucket-issue1880 Key: teststack-issue1185-cloudformation-637886578309076380.yaml) ... Progress: 100% Found existing stack: False CloudFormation change set created ... Waiting for change set to be reviewed Created CloudFormation stack teststack-issue1185

Timestamp Logical Resource Id Status

5/20/2022 3:37 PM teststack-issue1185 CREATE_IN_PROGRESS 5/20/2022 3:37 PM LambdaExecutionIamRole CREATE_IN_PROGRESS 5/20/2022 3:37 PM LambdaExecutionIamRole CREATE_IN_PROGRESS 5/20/2022 3:37 PM LambdaExecutionIamRole CREATE_COMPLETE 5/20/2022 3:37 PM AspNetCoreFunction CREATE_IN_PROGRESS 5/20/2022 3:37 PM AspNetCoreFunction CREATE_IN_PROGRESS 5/20/2022 3:37 PM AspNetCoreFunction CREATE_COMPLETE 5/20/2022 3:37 PM ServerlessApi CREATE_IN_PROGRESS 5/20/2022 3:37 PM ServerlessApi CREATE_IN_PROGRESS 5/20/2022 3:37 PM ServerlessApi CREATE_COMPLETE 5/20/2022 3:37 PM AspNetCoreFunctionRootResourcePermissionProd CREATE_IN_PROGRESS 5/20/2022 3:37 PM AspNetCoreFunctionProxyResourcePermissionProd CREATE_IN_PROGRESS 5/20/2022 3:37 PM ServerlessApiDeploymentcfb7a37fc3 CREATE_IN_PROGRESS 5/20/2022 3:37 PM AspNetCoreFunctionProxyResourcePermissionProd CREATE_IN_PROGRESS 5/20/2022 3:37 PM AspNetCoreFunctionRootResourcePermissionProd CREATE_IN_PROGRESS 5/20/2022 3:37 PM ServerlessApiDeploymentcfb7a37fc3 CREATE_IN_PROGRESS 5/20/2022 3:37 PM ServerlessApiDeploymentcfb7a37fc3 CREATE_COMPLETE 5/20/2022 3:37 PM ServerlessApiProdStage CREATE_IN_PROGRESS 5/20/2022 3:37 PM ServerlessApiProdStage CREATE_IN_PROGRESS 5/20/2022 3:37 PM ServerlessApiProdStage CREATE_COMPLETE 5/20/2022 3:38 PM AspNetCoreFunctionProxyResourcePermissionProd CREATE_COMPLETE 5/20/2022 3:38 PM AspNetCoreFunctionRootResourcePermissionProd CREATE_COMPLETE 5/20/2022 3:38 PM teststack-issue1185 CREATE_COMPLETE Stack finished updating with status: CREATE_COMPLETE


- Test the changes (this works fine)
<img width="773" alt="Screen Shot 2022-05-20 at 3 43 39 PM" src="https://user-images.githubusercontent.com/67916761/169621608-c78f34f8-aa13-4f4c-8b43-7b5a6c0b8710.png">

Executing `dotnet --version` gives output `6.0.202`.

Please let me know if I'm missing any step.

Thanks,
Ashish
madmox commented 2 years ago

Oh sorry, I made a last-minute change to my reproduction sample and it seems to be related to the buggy behavior! I updated the repo, just deployed the sample, and could reproduce the issue again.

It seems the bug has to do with the following lines:

context.Response.Body.Seek(0, SeekOrigin.Begin);
await context.Response.Body.CopyToAsync(originalResponseBodyStream);
context.Response.Body = originalResponseBodyStream;

If I replace this section with the following, it works as expected:

buffer.Seek(0, SeekOrigin.Begin);
await buffer.CopyToAsync(originalResponseBodyStream);
context.Response.Body = originalResponseBodyStream;

Although buffer and context.Response.Body are supposed to be the same object! (context.Response.Body = buffer)