OutOfMemoryException under high loads

[dpwilson]

My company is using RazorEngine to generate dynamic emails for our users and it has been working very well. However, we recently moved several high volume emails to the new template system and started running into an issue where the Windows service that handles the templating runs out of memory.

We have multiple Windows services pulling messages off a queue, generating the emails using RazorEngine and then sending the messages to an smtp server. We are currently sending 1+ million emails per day. At the moment we are using Razor.Parse with a cache name, keyed off the ID of the email type to generate both the subject line and body of the email. I have also profiled the app using Razor.Compile + Razor.Run, but we still run out of memory.

var model = emailRequest.MessageAttributes.ToExpando();

string emailSubject = Razor.Parse(emailTemplate.SubjectLine, model, "subject_" + emailRequest.EmailType).Replace(Environment.NewLine, " ");
string htmlContent = Razor.Parse(emailTemplate.Html, model, "html_" + emailRequest.EmailType).Replace(Environment.NewLine, "<br>");

string emailBody = emailTemplate.Master != null ? Razor.Parse(emailTemplate.Master, new { Content = htmlContent }, "master_" + emailRequest.EmailType) : htmlContent;

emailRequest.MessageAttributes is of type Dictionary<string,object> .ToExpando is an extension method that converts the dictionary to an Expando object

I have profiled the application and it seems that the dynamic models are being added to a Dictionary and never released. The error is thrown when the dictionary attempts to resize.

We have found that restarting the Windows service will clear the memory, but this is ideally a short term fix. I am trying to find the source of the memory leak and wondering how to flush the Dictionary without restarting the entire application.

RazorEngine.Razor holds onto a static instance of the TemplateService which then creates the TemplateServiceConfiguration and subsequently the CompilerService. I am trying to figure out if it is possible to Dispose of one of these classes to release the the references to the dynamic models.

The error message:

System.Collections.Generic.Dictionary`2.Resize(Int32 newSize, Boolean forceNewHashCodes)
   at System.Collections.Generic.Dictionary`2.Resize()
   at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
   at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
   at Microsoft.CSharp.RuntimeBinder.Semantics.SYMTBL.InsertChildNoGrow(Symbol child)
   at Microsoft.CSharp.RuntimeBinder.Semantics.SYMTBL.InsertChild(ParentSymbol parent, Symbol child)
   at Microsoft.CSharp.RuntimeBinder.Semantics.SymFactoryBase.newBasicSym(SYMKIND kind, Name name, ParentSymbol parent)
   at Microsoft.CSharp.RuntimeBinder.Semantics.SymFactory.CreateLocalVar(Name name, ParentSymbol parent, CType type)
   at Microsoft.CSharp.RuntimeBinder.RuntimeBinder.PopulateLocalScope(DynamicMetaObjectBinder payload, Scope pScope, ArgumentObject[] arguments, IEnumerable`1 parameterExpressions, Dictionary`2 dictionary)
   at Microsoft.CSharp.RuntimeBinder.RuntimeBinder.BindCore(DynamicMetaObjectBinder payload, IEnumerable`1 parameters, DynamicMetaObject[] args, DynamicMetaObject& deferredBinding)
   at Microsoft.CSharp.RuntimeBinder.RuntimeBinder.Bind(DynamicMetaObjectBinder payload, IEnumerable`1 parameters, DynamicMetaObject[] args, DynamicMetaObject& deferredBinding)
   at Microsoft.CSharp.RuntimeBinder.BinderHelper.Bind(DynamicMetaObjectBinder action, RuntimeBinder binder, IEnumerable`1 args, IEnumerable`1 arginfos, DynamicMetaObject onBindingError)
   at Microsoft.CSharp.RuntimeBinder.CSharpGetMemberBinder.FallbackGetMember(DynamicMetaObject target, DynamicMetaObject errorSuggestion)
   at System.Dynamic.GetMemberBinder.FallbackGetMember(DynamicMetaObject target)
   at System.Dynamic.ExpandoObject.MetaExpando.BindGetMember(GetMemberBinder binder)
   at System.Dynamic.GetMemberBinder.Bind(DynamicMetaObject target, DynamicMetaObject[] args)
   at System.Dynamic.DynamicMetaObjectBinder.Bind(Object[] args, ReadOnlyCollection`1 parameters, LabelTarget returnLabel)
   at System.Runtime.CompilerServices.CallSiteBinder.BindCore[T](CallSite`1 site, Object[] args)
   at System.Dynamic.UpdateDelegates.UpdateAndExecute1[T0,TRet](CallSite site, T0 arg0)
   at CallSite.Target(Closure , CallSite , Object )
   at CompiledRazorTemplates.Dynamic.dfacedbdffaaf.Execute()
   at RazorEngine.Templating.TemplateBase.RazorEngine.Templating.ITemplate.Run(ExecuteContext context) in c:\Users\abbottm\Documents\GitHub\RazorEngine\src\Core\RazorEngine.Core\Templating\TemplateBase.cs:line 126
   at RazorEngine.Templating.TemplateService.Run(ITemplate template, DynamicViewBag viewBag) in c:\Users\abbottm\Documents\GitHub\RazorEngine\src\Core\RazorEngine.Core\Templating\TemplateService.cs:line 608
   at RazorEngine.Templating.TemplateService.Run(String cacheName, Object model, DynamicViewBag viewBag) in c:\Users\abbottm\Documents\GitHub\RazorEngine\src\Core\RazorEngine.Core\Templating\TemplateService.cs:line 594
   at RazorEngine.Razor.Run[T](String cacheName, T model) in c:\Users\abbottm\Documents\GitHub\RazorEngine\src\Core\RazorEngine.Core\Razor.cs:line 593

[dpwilson]

We have changed our implementation to pass a DynamicViewBag instead of an ExpandoObject and this resulted in an extreme improvement in the memory usage of the application. (We also updated our templates with "@ViewBag.")

DynamicViewBag viewBag = new DynamicViewBag();
viewBag.AddDictionaryValues(emailRequest.MessageAttributes);

string emailSubject = Razor.Parse(emailTemplate.SubjectLine, null, viewBag, "subject_" + emailRequest.EmailType).Replace(Environment.NewLine, " ");

Our theory is that during the process of "hydrating" the template the dynamic expando object is converted into a strongly typed object and the reference is never released. By passing through the ViewBag .Net Razor can treat the object dynamically and will not hold onto the reference. I can't say for sure if this is what is happening, but it lines up with the results we are seeing when profiling the memory usage.

This doesn't resolve the memory issue when passing a dynamic model, but it should provide an alternative when using objects that are not strongly typed.

[matthid]

Any chance you where hit by: https://connect.microsoft.com/VisualStudio/feedback/details/783541/expandoobjects-leak-memory-when-used-as-an-idictionary-string-object-to-add-new-properties? This doesn't look like a RazorEngine specific thing.

Edit: Out of curiosity I could look closer into this, any chance of providing a test-case?