FunctionInvocationException: Failed to proxy request with ForwarderError: RequestBodyDestination

[ipetrouchtchak-fi]

I have a HttpTrigger function, which reads from a File part in multipart/form-data. The function was just converted from the in-process model (where it didn't have any issues with uploading large files ~200 MB).

In the .NET 8.0 Isolated model, when attempting to upload files larger than about 30 MB, a 500 error is returned.

Error:

location: West US 3
...
Microsoft.Azure.WebJobs.Host.FunctionInvocationException : Exception while executing function: Functions.DocUpload 
---> System.InvalidOperationException : Failed to proxy request with ForwarderError: RequestBodyDestination 
---> System.IO.IOException : Unable to write data to the transport connection: An established connection was aborted by the software in your host machine.. 
---> System.Net.Sockets.SocketException : An established connection was aborted by the software in your host machine.

   End of inner exception
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async System.Net.Http.HttpConnection.<WriteAsync>g__AwaitFlushAndWriteAsync|72_0(??)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async System.Net.Http.HttpConnection.ChunkedEncodingWriteStream.<WriteAsync>g__WriteChunkAsync|4_0(??)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Yarp.ReverseProxy.Forwarder.StreamCopier.CopyAsync(??)
   End of inner exception
   at async Microsoft.Azure.WebJobs.Script.Grpc.DefaultHttpProxyService.EnsureSuccessfulForwardingAsync(ScriptInvocationContext context) at /_/src/WebJobs.Script.Grpc/Server/DefaultHttpProxyService.cs : 65
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Script.Grpc.GrpcWorkerChannel.InvokeResponse(InvocationResponse invokeResponse) at /_/src/WebJobs.Script.Grpc/Channel/GrpcWorkerChannel.cs : 1116
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Script.Description.WorkerFunctionInvoker.InvokeCore(Object[] parameters,FunctionInvocationContext context) at /_/src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs : 101
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) at /_/src/WebJobs.Script/Description/FunctionInvokerBase.cs : 82
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Script.Description.FunctionGenerator.Coerce[T](Task`1 src) at /_/src/WebJobs.Script/Description/FunctionGenerator.cs : 225
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.InvokeAsync[TReflected,TReturnValue](Object instance,Object[] arguments) at D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs : 53
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeWithTimeoutAsync(IFunctionInvoker invoker,ParameterHelper parameterHelper,CancellationTokenSource timeoutTokenSource,CancellationTokenSource functionCancellationTokenSource,Boolean throwOnTimeout,TimeSpan timerInterval,IFunctionInstance instance) at D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 581
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstanceEx instance,ParameterHelper parameterHelper,ILogger logger,CancellationTokenSource functionCancellationTokenSource) at D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 523
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance,FunctionStartedMessage message,FunctionInstanceLogEntry instanceLogEntry,ParameterHelper parameterHelper,ILogger logger,CancellationToken cancellationToken) at D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 306
   End of inner exception
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance,FunctionStartedMessage message,FunctionInstanceLogEntry instanceLogEntry,ParameterHelper parameterHelper,ILogger logger,CancellationToken cancellationToken) at D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 352
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync(IFunctionInstance functionInstance,CancellationToken cancellationToken) at D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 108

...

     exceptionType: System.Net.Sockets.SocketException
     functionInvocationId: 125a0f8f-8e66-4de7-8e92-64cfed739bcf
     hostVersion: 4.1036.3.23284
     level: Error
     roleInstance: 063162454eb26db9d029efded3832739f5a496e8329bffdfa4913863a9d915b9

FileSystem Logs:

Exception: Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Request body too large. The max request body size is 30000000 bytes.
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ChunkedEncodingMessageBody.Read(ReadOnlySequence`1 readableBuffer, PipeWriter writableBuffer, SequencePosition& consumed, SequencePosition& examined)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ChunkedEncodingMessageBody.PumpAsync()
at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)at System.IO.Pipelines.Pipe.GetReadAsyncResult()
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ChunkedEncodingMessageBody.ReadAsyncInternal(CancellationToken cancellationToken)at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.ReadAsyncInternal(Memory`1 destination, CancellationToken cancellationToken)
at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)at Microsoft.AspNetCore.WebUtilities.BufferedReadStream.EnsureBufferedAsync(Int32 minCount, CancellationToken cancellationToken)at Microsoft.AspNetCore.WebUtilities.MultipartReaderStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken)at Microsoft.AspNetCore.WebUtilities.FileBufferingReadStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken)at Microsoft.AspNetCore.WebUtilities.StreamHelperExtensions.DrainAsync(Stream stream, ArrayPool`1 bytePool, Nullable`1 limit, CancellationToken cancellationToken)at Microsoft.AspNetCore.Http.Features.FormFeature.InnerReadFormAsync(CancellationToken cancellationToken)
...

[bhagyshricompany]

Hi @ipetrouchtchak-fi for Thanks for reporting please share the all repro steps and file sample..Thanks

[ipetrouchtchak-fi]

Hi Thanks for reporting please share the all repro steps and file sample.

@bhagyshricompany, here are the basic steps:

1) In a .NET 8.0 Isolated Function App (Premium v2 P1V2 App Service Plan) 2) Create Http Triggered function with a post and HttpRequest req, which reads Form Data from the request

formData = await req.ReadFormAsync();

3) Post a multipart/form-data with a File larger than 30000000 bytes