AWS Lambda in .net 6.0 create some files in temp storage when it fails with "Exceeded maximum allowed payload size" exception

[asn1997]

Describe the bug

AWS Lambda in .net 6.0 create some files in temp storage when it fails with "Exceeded maximum allowed payload size" exception. It create folowing file "/tmp/core.dotnet.8" and it will not be deleted and this impact the next execution in same Lambda Instance.

We use NewtonsoftJson as Input and Output formator so it use disk space internally when the data is more than 30KB.

So if we invoke Lambda after this error "Exceeded maximum allowed payload size" and the request goes to same instance and if the response is more than 30 KB it tries to create a temp file in /tmp folder then it throws this error

"ClassName":"System.IO.IOException","Message":"No space left on device : '/tmp/ASPNETCORE_5a294445-5cfd-4db9-8049-d2ef44559710.tmp'","Data":{},"InnerException":null,"HelpURL":null,"StackTraceString":" at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan1 buffer, Int64 fileOffset)\n at System.IO.Strategies.OSFileStreamStrategy.Write(Byte[] buffer, Int32 offset, Int32 count)\n at Microsoft.AspNetCore.WebUtilities.PagedByteBuffer.MoveTo(Stream stream)\n at Microsoft.AspNetCore.WebUtilities.FileBufferingWriteStream.Write(Byte[] buffer, Int32 offset, Int32 count)\n at Microsoft.AspNetCore.WebUtilities.HttpResponseStreamWriter.Write(String value)\n at Newtonsoft.Json.Utilities.JavaScriptUtils.WriteEscapedJavaScriptString(TextWriter writer, String s, Char delimiter, Boolean appendDelimiters, Boolean[] charEscapeFlags, StringEscapeHandling stringEscapeHandling, IArrayPool1 bufferPool, Char[]& writeBuffer)\n at Newtonsoft.Json.JsonTextWriter.WritePropertyName(String name, Boolean escape)\n at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeDictionary(JsonWriter writer, IDictionary values, JsonDictionaryContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)\n at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeList(JsonWriter writer, IEnumerable values, JsonArrayContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)\n at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeDictionary(JsonWriter writer, IDictionary values, JsonDictionaryContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)\n at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType)\n at Newtonsoft.Json.JsonSerializer.SerializeInternal(JsonWriter jsonWriter, Object value, Type objectType)\n at Microsoft.AspNetCore.Mvc.Formatters.NewtonsoftJsonOutputFormatter.WriteResponseBodyAsync(OutputFormatterWriteContext context, Encoding selectedEncoding)\n at Microsoft.AspNetCore.Mvc.Formatters.NewtonsoftJsonOutputFormatter.WriteResponseBodyAsync(OutputFormatterWriteContext context, Encoding selectedEncoding)\n at Microsoft.AspNetCore.Mvc.Formatters.NewtonsoftJsonOutputFormatter.WriteResponseBodyAsync(OutputFormatterWriteContext context, Encoding selectedEncoding)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.gAwaited|30_0[TFilter,TFilterAsync](ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResultExecutedContextSealed context)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ResultNext[TFilter,TFilterAsync](State& next, Scope& scope, Object& state, Boolean& isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeResultFilters()\n--- End of stack trace from previous location ---\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.gAwaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.gAwaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)\n at Microsoft.AspNetCore.Routing.EndpointMiddleware.gAwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)\n at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)","RemoteStackTraceString":null,"RemoteStackIndex":0,"ExceptionMethod":null,"HResult":28,"Source":"System.Private.CoreLib","WatsonBuckets":null

this happens because disk space is full because of this file "/tmp/core.dotnet.8"

Expected Behavior

Temp files like "/tmp/core.dotnet.8" which got created after this exception "Exceeded maximum allowed payload size" should be deleted

Current Behavior

Temp files like "/tmp/core.dotnet.8" which got created after this exception "Exceeded maximum allowed payload size" are not deleted

Reproduction Steps

Invoke the Lambda to return response which is greater than 6MB 3 to 4 times in a way that all request should go to the same instance of the Lambda, Then invoke the lambda with request or request more than 30 KB this error will occor

Possible Solution

Temp files like "/tmp/core.dotnet.8" which got created after this exception "Exceeded maximum allowed payload size" should be deleted

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

AWSSDK.Core 3.7.107.8

Targeted .NET Platform

.NET 6

Operating System and version

AWS Lambda AmazonLinux x86_64

[ashishdhingra]

@asn1997 Good morning. Could you please share the below details:

• Sample code to reproduce the issue?
• How is Lambda executed? Is it via API Gateway?
• Please refer Lambda quotas. It specifies that the maximum invocation payload is 6 MB each for request and response (synchronous). Looks like you are exceeding the quota.
  • As also in Lambda quotas, the quotas apply to function configuration, deployment, and execution. Except as noted, they can't be changed..
• I noticed that you also opened discussion https://github.com/aws/aws-lambda-dotnet/discussions/1537 for this issue and that you migrated it to 6.0 from .net 3.1. Did the payload size changed after you migrated or was it working with the same payload size (greater than 6MB) when using .NET 3.1 runtime? (I will close the discussion to track it as part of this issue). Since the quotas cannot be changed and it was working earlier, you might be exceeding quota after the upgrade to .NET 6.

• Also refer https://repost.aws/questions/QU2D7iduzHSVWNlLn69ItyJw/increase-quota-synchronous-payload-on-aws-lambda-to-receive-large-video-file for the similar issue. It might not be possible to increase the payload size to greater than 6MB as in your use case.

Thanks, Ashish

[asn1997]

• How is Lambda executed? Is it via API Gateway? -> Yes it is via API Gateway

We are aware of the Quotas but it should not create some temp file /tmp folder when Lambda try to return data more than 6 MB.

[asn1997]

Did the payload size changed after you migrated or was it working with the same payload size (greater than 6MB) when using .NET 3.1 runtime? -> No

[ashishdhingra]

@asn1997 Could you please code (preferably solution for easy reproduction) to reproduce the issue? There is a similar closed issue https://github.com/dotnet/aspnetcore/issues/22342 in Microsoft's .NET forums, unsure if the same issue has surfaced back in .NET 6.

[asn1997]

@ashishdhingra https://github.com/dotnet/aspnetcore/issues/22342 in this issue it is mention NewtonsoftJsonInputFormatter creates File when input size is more than 30KB and that file name will be like ASPNETCORE_guid.tmp.

But i am not seeing any file is there in /tmp folder with this kind of name, in my lambda i have printed what files are there in /tmpfolder and these are the files with size:

{ "/tmp/core.dotnet.8": 734330880, "/tmp/dotnet-diagnostic-7-584793-socket": 0, "/tmp/clr-debug-pipe-7-584793-in": 0, "/tmp/core.dotnet.9": 559923200, "/tmp/clr-debug-pipe-7-584793-out": 0 }

This Lamda we use as our Backend API so if user want to retrieve large data which is more tha 6 MB then only these kind of files are being created. if we are getting a error of 6 MB limit that is not the issue. But it should not create any temp files, because these files will be deleted only when that particular instance of that lambda gets killed, 1 lambda instance can be available upto 5 mins and if any other request come in mean time it can go to the same instance.

[asn1997]

@ashishdhingra did you get any solution for this?

[asn1997]

Hi @ashishdhingra any update on this?

[ashishdhingra]

@asn1997 Could you please share sample code solution to reproduce the issue?

[StewartSnow]

We have the same issue - pretty much identical to this issue: https://github.com/dotnet/aspnetcore/issues/48149

Except for us, we're not writing anything to the disk at all - something is happening that's filling it up and then the code fails with the exact error described in the linked issue.

It's hard to reproduce as it does it sporadically. And once it's occurred on a lambda container - the container continues to serve up errors anytime the code (NewtsoftJson) tries to write anything to disk... So the error makes it look like it's Newtonsoft, but we suspect that this is a symptom of the issue, not the cause.

Note this has been occurring for us in APIs running within AWS Lambda specifically since upgrading to .net6 - prior to this it wasn't an issue.

We've extended the lambda storage up to 2048MB but it still occurs sporadically...

[ashishdhingra]

@asn1997 Could you please check if this is still the issue with latest Lambda .NET 6 runtime?

[github-actions[bot]]

This issue has not received a response in 5 days. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled.

[asn1997]

Hi @ashishdhingra ,

We still have this issue for .net 6 AWS Lambda

[asn1997]

I want to Reopen this issue I am still facing this issue, Please Reopen it @ashishdhingra

[wr-cdargis]

For what it's worth: IMO the biggest problem with this is that if a lambda instance experiences this error it is still continues to be reused on subsequent requests even though the disk space is still occupied. This causes all subsequent requests on this instance to fail. You have to wait until the lambda instance falls out of rotation and AWS invokes the shutdown event. We were able to mitigate this problem by catching this error via middleware and cleaning up the disk space. As far as what causes it in the first place I believe it's an issue with paging in a very large amount of data in the request/response.

[asn1997]

We were able to mitigate this problem by catching this error via middleware and cleaning up the disk space. As far as what causes it in the first place I believe it's an issue with paging in a very large amount of data in the request/response

Yes Exactly the same issue we are facing

[asn1997]

We get this error from lambda instance which already failed with this error "Exceeded maximum allowed payload size" then any other request which goes to same instance and try to retrieve large amount of data but less than 6MB, fails with this error "System.IO.IOException","Message":"No space left on device"

[StewartSnow]

Has anyone established if the problem also exists under .net 8? We're still on 6 and have the problem sporadically, but if 8 resolves it then an easy fix etc.

[WillCallahan]

This is still an issue in .NET 6

[VinzentJenner]

The issue remains for .NET 8.