search

toddams / RazorLight

Template engine based on Microsoft's Razor parsing engine for .NET Core
Apache License 2.0
1.5k stars 259 forks source link

Out Of Memory Exception while converting large model to html string using RenderTemplateAsync #453

Closed PraveenMFluke closed 2 years ago

PraveenMFluke commented 2 years ago

Describe the bug We are using RazorLightEngine to convert our Model to Html string using "RenderTemplateAsync" method of RazorLightEngine . Our Model is Dictionary of classes and this collection contains hundred thousands of entries( 300000 entries in our case). Once we recieve the output of the method "RenderTemplateAsync". which is a string , we pass this to pdf renderer class to get the bytes for pdf generation . However while converting our model to string using the template "RenderTemplateAsync" throws Out Of Memory Exception . Our html template is merely a table structure . Let me know if you have any solution to this . Or If this API supports any batch processing since the amount of data we are supplying is huge .

To Reproduce Have a model with hundred thousands of entries and pass it to "RenderTemplateAsync" and it throws Out Of Memory Exception

Expected behavior RenderTemplateAsync should return string by using the template and model we provide

Information (please complete the following information):

jzabroski commented 2 years ago

@PraveenMFluke Can you be more specific? I can create an OutOfMemoryException just by calling string.Empty.PadLeft(' ', long.MaxValue) over and over again, so I am not sure what is causing your out of memory exception. Given you are also using a third party library for pdf rendering, I suggest you use a tool like RedGate ANTS+ Memory Profiler and look at which objects have the most pooled memory at the time you get the OutOfMemoryException. It's possible you are not disposing of something you should.

Or If this API supports any batch processing since the amount of data we are supplying is huge .

The rendered template itself is not a disposable stream. It's a string, so it can be at most 2GB realistically. Can you do a back of the napkin calculating on the sizeof your data structure and how much GB your 300,000 entries take up? Also, if this OOM during an application run, or during a unit test, because the unit test runner itself may be running out of memory prior to the string allocation.

PraveenMFluke commented 2 years ago

I have quickly checked the object size . It was around 900MB for 300,000 entries before passing to RenderTemplateAsync . Basically our object is IEnumerable of Dictionary which will be filled after reading DB . This is ASP.NET background application . Not found during unit testing

jzabroski commented 2 years ago

Can you please not "quickly check" a performance issue and submit a bug report for it? If you want help - prove the issue is RazorLight. When you say it's an ASP.NET background application - what does that mean? There are no ASP.NET background applications. If you're using .NET 4.8, you have the queue a background item, and that background item has to complete within 5 minutes.

jzabroski commented 2 years ago

@PraveenMFluke The only advice I can give is to make sure you disable caching. But I think before you go making changes you really need to pull out a performance profiler and determine what is using up memory.