NancyFx / Nancy

Lightweight, low-ceremony, framework for building HTTP based services on .Net and Mono
http://nancyfx.org
MIT License
7.16k stars 1.47k forks source link

Large content length uploads (>32 bit) are loaded into memory, rather than using fs cache #2983

Open BTMorton opened 4 years ago

BTMorton commented 4 years ago

Prerequisites

Description

When uploading a large file using Nancy (>2GB) I get a "Stream was too long" error as it looks like the entire upload request stream is attempting to be loaded into memory.

fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HLPQTF9FBMQ6", Request id "0HLPQTF9FBMQ6:00000001": An unhandled exception was thrown by the application.
System.IO.IOException: Stream was too long.
   at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.Stream.CopyTo(Stream destination, Int32 bufferSize)
   at Nancy.IO.RequestStream..ctor(Stream stream, Int64 expectedLength, Int64 thresholdLength, Boolean disableStreamSwitching)
   at Nancy.Owin.NancyMiddleware.<>c__DisplayClass2_1.<<UseNancy>b__1>d.MoveNext()

I've narrowed this down to the NancyMiddleware class, ExpectedLength method. It's has a long return type but is parsing the content length string using int:

return int.TryParse(header, NumberStyles.Any, CultureInfo.InvariantCulture, out int contentLength) ? contentLength : (long?)null;

This cannot parse a long value and returns null, which passes an expected length of 0 to RequestStream which causes it to try to read the whole upload stream into memory.

I'm not sure if this is intentional or just overlooked, but I figured I would raise an issue before opening a pull request. Checking out Nancy locally and building with long.TryParse works correctly (despite being slow).

Steps to Reproduce

Run a Nancy sample app. POST a > 2GB file to any endpoint. Get 500 response and see "Stream was too large" error in console.

System Configuration

Running in Visual Studio 2019 and uploading a 4gb file with postman (doesn't currently support chunked transfer encoding).