praeclarum / FuGetGallery

An alternative web UI for browsing nuget packages
https://www.fuget.org
MIT License
685 stars 121 forks source link

Out of memory while viewing specific packages #48

Open vinodmishra opened 6 years ago

vinodmishra commented 6 years ago

Going to https://www.fuget.org/packages/NHibernate results in Error reading package System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown. at System.IO.MemoryStream.set_Capacity(Int32 value) at System.IO.MemoryStream.EnsureCapacity(Int32 value) at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count) at System.IO.MemoryStream.CopyToAsync(Stream destination, Int32 bufferSize, CancellationToken cancellationToken) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at FuGetGallery.PackageData.PackageDataCache.<GetValueAsync>d__2.MoveNext() in D:\home\site\repository\Data\PackageData.cs:line 314

Works fine for older versions of nhibernate https://www.fuget.org/packages/nhibernate/5.0.6/lib

praeclarum commented 6 years ago

Ugh, it seems to be a temporary bug. I will look into that code path and see if there's a way to not allocate the whole buffer.

MatthewSteeples commented 5 years ago

I've got 2 suggestions looking at the location of the stacktrace 1) The first one is to dispose the memory stream once finished with it (in a using statement) 2) The second is to have a look at https://github.com/Microsoft/Microsoft.IO.RecyclableMemoryStream instead of using MemoryStream.

The reason for number 2 is that you tend to find that these exceptions aren't to do with running out of memory, but are to do with running out of a block of memory large enough to fit the array in one go. Using pooling can help alleviate this.

If you don't want to add a third party dependency, then initialising the MemoryStream to be the correct size to start with (based on the Http response) will also help reduce the number of occurrences of this

mjkkirschner commented 5 years ago

I see a similar exception:

System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.IO.MemoryStream.set_Capacity(Int32 value)
   at System.IO.MemoryStream.EnsureCapacity(Int32 value)
   at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.MemoryStream.CopyToAsync(Stream destination, Int32 bufferSize, CancellationToken cancellationToken)
--- End of stack trace from previous location where exception was thrown ---
   at FuGetGallery.PackageData.PackageDataCache.ReadPackageFromUrl(PackageData package, HttpClient httpClient, CancellationToken token) in D:\home\site\repository\Data\PackageData.cs:line 405
   at FuGetGallery.PackageData.PackageDataCache.GetValueAsync(String arg0, PackageVersion arg1, HttpClient httpClient, CancellationToken token) in D:\home\site\repository\Data\PackageData.cs:line 381
Dependents 

https://www.fuget.org/packages/DynamoVisualProgramming.Core

older versions which used to work and are cached on my machine - now seem to throw this exception when I try to reach them in an incognito window. update I built locally and cannot reproduce - and of course now when I try the same link on fuget it loads super fast. 😕 (which is what I guess was meant about temporary error above)

praeclarum commented 4 years ago

@MatthewSteeples that's a cool class, I'll check it out!

@marcinsmialek Yeah I'm pretty careless with my memory management and probably have similar bugs. I will try to switch to pooling or sequences to help. My guess is that other things be retained in memory too long (e.g. I need to look into the cache more) are contributing.

jzabroski commented 4 years ago

108 appears to be a duplicate of this issue