ImageSharp System.OutOfMemoryException After v13 Upgrade

joshanangelofgrace commented 6 months ago

Which Umbraco version are you using? (Please write the exact version, example: 10.1.0)

13.3.0

Bug summary

Over the weekend we upgraded out site from v12 to v13 and noticed that we were getting System.OutOfMemoryException from ImageSharp. We did some testing and noticed that one particular image was causing the error. Once we changed the image it fixed it. However we are wanting to bring your awareness to the issue.

Raising ticket on behalf of YourITTeam (AU)

Specifics

URL: https://kingfisher-fiber.useast01.umbraco.io/media/2992/office-logo.png?width=1200&height=630&rnd=133267957852700000

Exception: Insufficient memory to continue the execution of the program. System.Runtime.InteropServices.Marshal.AllocHGlobal(IntPtr cb):17 SixLabors.ImageSharp.Memory.Internals.UnmanagedMemoryHandle.AllocateHandle(Int32 lengthInBytes):11 SixLabors.ImageSharp.Memory.Internals.UnmanagedMemoryHandle.Allocate(Int32 lengthInBytes) SixLabors.ImageSharp.Memory.Internals.UnmanagedBuffer1.Allocate(Int32 lengthInElements) SixLabors.ImageSharp.Memory.UnmanagedMemoryAllocator.Allocate[T](Int32 length, AllocationOptions options) SixLabors.ImageSharp.Memory.UniformUnmanagedMemoryPoolMemoryAllocator.Allocate[T](Int32 length, AllocationOptions options):112 SixLabors.ImageSharp.Formats.Png.PngDecoderCore.ReadChunkData(Int32 length):14 SixLabors.ImageSharp.Formats.Png.PngDecoderCore.TryReadChunk(Span1 buffer, PngChunk& chunk):208 SixLabors.ImageSharp.Formats.Png.PngDecoderCore.Decode[TPixel](BufferedReadStream stream, CancellationToken cancellationToken):72 SixLabors.ImageSharp.Formats.ImageDecoderUtilities.Decode[TPixel](IImageDecoderInternals decoder, Configuration configuration, Stream stream, Func3 largeImageExceptionFactory, CancellationToken cancellationToken):20 SixLabors.ImageSharp.Formats.ImageDecoderUtilities.Decode[TPixel](IImageDecoderInternals decoder, Configuration configuration, Stream stream, CancellationToken cancellationToken):15 SixLabors.ImageSharp.Formats.Png.PngDecoder.Decode[TPixel](PngDecoderOptions options, Stream stream, CancellationToken cancellationToken):28 SixLabors.ImageSharp.Formats.Png.PngDecoder.Decode(PngDecoderOptions options, Stream stream, CancellationToken cancellationToken):334 SixLabors.ImageSharp.Formats.SpecializedImageDecoder1.Decode(DecoderOptions options, Stream stream, CancellationToken cancellationToken) SixLabors.ImageSharp.Formats.ImageDecoder+<>cDisplayClass3_0.b0(Stream s, CancellationToken ct) SixLabors.ImageSharp.Formats.ImageDecoder+<>cDisplayClass12_0`1.gPeformActionAndResetPosition|0(Stream s, Int64 position, CancellationToken ct) System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw():17 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task):55 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task, ConfigureAwaitOptions options):45 System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter.GetResult()

SixLabors.ImageSharp.Formats.ImageDecoder+d__131.MoveNext():339 System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw():17 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task):55 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task, ConfigureAwaitOptions options):45 System.Runtime.CompilerServices.ConfiguredTaskAwaitable1+ConfiguredTaskAwaiter.GetResult()

SixLabors.ImageSharp.Formats.ImageDecoder+d__3.MoveNext():198 System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw():17 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task):55 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task, ConfigureAwaitOptions options):45 System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter.GetResult()

SixLabors.ImageSharp.Image+d__881.MoveNext():238 System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw():17 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task):55 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task, ConfigureAwaitOptions options):45 System.Runtime.CompilerServices.TaskAwaiter1.GetResult()

SixLabors.ImageSharp.Web.FormattedImage+d__16.MoveNext():111 System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw():17 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task):55 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task, ConfigureAwaitOptions options):45 System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()

SixLabors.ImageSharp.Web.Middleware.ImageSharpMiddleware+d21.MoveNext():1666 System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw():17 SixLabors.ImageSharp.Web.Middleware.ImageSharpMiddleware+d21.MoveNext():2718 System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw():17 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task):55 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task, ConfigureAwaitOptions options):45

SixLabors.ImageSharp.Web.Middleware.ImageSharpMiddleware+d__19.MoveNext():1201 System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw():17 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task):55 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task, ConfigureAwaitOptions options):45

Umbraco.Forms.Web.HttpModules.ProtectFormUploadRequestsMiddleware+d__4.MoveNext():301 System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw():17 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task):55 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task, ConfigureAwaitOptions options):45

Microsoft.AspNetCore.Builder.UseMiddlewareExtensions+InterfaceMiddlewareBinder+<>c__DisplayClass2_0+<b__0>d.MoveNext():260 System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw():17 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task):55 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task, ConfigureAwaitOptions options):45

Umbraco.Commerce.Cms.Web.Mvc.UmbracoCommerceRequestBufferingMiddleware+d__0.MoveNext():171 System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw():17 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task):55 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task, ConfigureAwaitOptions options):45

Microsoft.AspNetCore.Builder.UseMiddlewareExtensions+InterfaceMiddlewareBinder+<>c__DisplayClass2_0+<b__0>d.MoveNext():260 System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw():17 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task):55 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task, ConfigureAwaitOptions options):45

Mindscape.Raygun4Net.AspNetCore.RaygunMiddleware+d__6.MoveNext() in C:\BuildAgent\work\75116afedfe31196\Mindscape.Raygun4Net.AspNetCore\RaygunMiddleware.cs:43

Steps to reproduce

This error was caused by this image

https://kingfisher-fiber.useast01.umbraco.io/media/05mjbhfi/office-logo-v2.png

We took a screenshot and then uploaded and the error was no longer generated

We suspect it was caused by the colour spectrum of the image

Expected result / actual result

Image causes no error when processing

github-actions[bot] commented 6 months ago

Hi there @joshanangelofgrace!

Firstly, a big thank you for raising this issue. Every piece of feedback we receive helps us to make Umbraco better.

We really appreciate your patience while we wait for our team to have a look at this but we wanted to let you know that we see this and share with you the plan for what comes next.

We'll assess whether this issue relates to something that has already been fixed in a later version of the release that it has been raised for.
If it's a bug, is it related to a release that we are actively supporting or is it related to a release that's in the end-of-life or security-only phase?
We'll replicate the issue to ensure that the problem is as described.
We'll decide whether the behavior is an issue or if the behavior is intended.

We wish we could work with everyone directly and assess your issue immediately but we're in the fortunate position of having lots of contributions to work with and only a few humans who are able to do it. We are making progress though and in the meantime, we will keep you in the loop and let you know when we have any questions.

Thanks, from your friendly Umbraco GitHub bot :robot: :slightly_smiling_face:

NguyenThuyLan commented 4 months ago

Hi @joshanangelofgrace , thanks for reporting this issue. Could you describe this issue more clearly? I need more details for Steps to reproduce. Actually, I used your image to upload to my project on v13, and my project on v12 and then upgraded to v13. It works as normal. So could you please describe more details for Steps To Reproduce?

keremkambur commented 2 days ago

ImageSharp is just killing the memory, I end up with 5 GBs of unmanaged memory out of 7 GB memory usage. Under these circumstances, Umbraco is really not a solution for relatively higher scale. Despite we cache the images on CDN, memory usage never calms down (we do not have frequent new images) It ramps up when images asked concurrently. We are using load balanced server setup with 2 Subscriber instances and 1 CMS instance. Attached memory profile result is from Subscriber profile. Umbraco version: v13.5.2

bergmania commented 2 days ago

Maybe consider if you are using the correct Garbage Collector Mode, for your setup. https://learn.microsoft.com/en-us/aspnet/core/performance/memory?view=aspnetcore-8.0#workstation-gc-vs-server-gc

Not sure what we can do about it - As you mention it is ImageSharp that behaves like this. If the default ImageSharp implementation do not work in your scenarios, you can replace it with your own implementation. By the end of the day, Umbraco just generate Urls to images, that needs to be handled - It could be handled by all kind of tools.

keremkambur commented 2 days ago

Thank you for the suggestion, I will play around with the GC mode and will try out replacing ImageSharp with Magick.NET though I couldn't find an official guide to do so.

I am aware that this is not something Umbraco CMS itself is causing but ImageSharp memory issue is haunting Umbraco since many versions I observed (from 11 to 13 at least). Hope there is an item on the roadmap to visit this issue, thank you for the great work you are putting on, apologies if my previous remark sounded unpleasant. Because this issue is incredibly affecting our server costs and has quite observable performance implications in certain periods.

keremkambur commented 2 days ago

@bergmania Unfortunately it was unrelated to the GC settings, I tried both ways on both desktop and server environments. ImageSharp contains lots of unmanaged code. Could you please share your idea or refer a post on how to replace it with another provider like Magick.NET? Thank you

bergmania commented 2 days ago

@keremkambur you will need to replace the implementation of IImageUrlGenerator and potentially IImageDimensionExtractor those both have ImageSharp implementations.

The IImageUrlGenerator basically just needs to generate a url. When that url is requested you will need to do roughly the same as ImageSharp.Web - Make some middleware that handles the request and return the image what is requested. You can use the parameters from the Url to get the info .e.g. width and height...

keremkambur commented 2 days ago

I have visited the interfaces and from your note, I understood the idea overall :slightly_smiling_face: Is it easily possible to register our own implementation just as using IComposer without touching the Umbraco CMS code? (i.e. a need of private NuGet package/pull request? Thank you

bergmania commented 2 days ago

Sure you register your own using IComposer! The Umbraco.Cms.Imaging.ImageSharp assemblies are doing the same, so you can do it from your solution. We do not wanna take a reference on Imagemagick, but you can do that from your solution

umbraco / Umbraco-CMS