Out of Memory Exception when rendering large history

[fireundubh]

I get an out of memory exception when rendering a very large channel history as HTML.

Here's the line where the exporter crashed in DiscordChatExporter.Core.Services.ExportService:

            // Render output
            using (var output = File.CreateText(filePath))
            {
                // Configure output
                context.PushOutput(new TextWriterOutput(output));

                // Render template
                template.Render(context); // <------------------
            }

Here's the stack:

System.OutOfMemoryException
  HResult=0x8007000E
  Source=mscorlib
  StackTrace:
   at System.Text.StringBuilder.set_Length(Int32 value)
   at Scriban.Template.Render(TemplateContext context)
   at Scriban.Functions.IncludeFunction.Invoke(TemplateContext context, ScriptNode callerContext, ScriptArray arguments, ScriptBlockStatement blockStatement)
   at Scriban.Syntax.ScriptFunctionCall.Call(TemplateContext context, ScriptNode callerContext, Object functionObject, Boolean processPipeArguments, List`1 arguments)
   at Scriban.Syntax.ScriptFunctionCall.Evaluate(TemplateContext context)
   at Scriban.TemplateContext.EvaluateImpl(ScriptNode scriptNode)
   at Scriban.TemplateContext.Evaluate(ScriptNode scriptNode, Boolean aliasReturnedFunction)
   at Scriban.Syntax.ScriptExpressionStatement.Evaluate(TemplateContext context)
   at Scriban.TemplateContext.EvaluateImpl(ScriptNode scriptNode)
   at Scriban.TemplateContext.Evaluate(ScriptNode scriptNode, Boolean aliasReturnedFunction)
   at Scriban.Syntax.ScriptBlockStatement.Evaluate(TemplateContext context)
   at Scriban.TemplateContext.EvaluateImpl(ScriptNode scriptNode)
   at Scriban.TemplateContext.Evaluate(ScriptNode scriptNode, Boolean aliasReturnedFunction)
   at Scriban.Syntax.ScriptPage.Evaluate(TemplateContext context)
   at Scriban.TemplateContext.EvaluateImpl(ScriptNode scriptNode)
   at Scriban.TemplateContext.Evaluate(ScriptNode scriptNode, Boolean aliasReturnedFunction)
   at Scriban.Template.EvaluateAndRender(TemplateContext context, Boolean render)
   at Scriban.Template.Render(TemplateContext context)
   at DiscordChatExporter.Core.Services.ExportService.Export(ExportFormat format, String filePath, ChatLog log) in E:\projects\DiscordChatExporter\DiscordChatExporter.Core\Services\ExportService.cs:line 53
   at DiscordChatExporter.Gui.ViewModels.MainViewModel.<Export>d__63.MoveNext() in E:\projects\DiscordChatExporter\DiscordChatExporter.Gui\ViewModels\MainViewModel.cs:line 273

Here's a related issue on the Scriban tracker: https://github.com/lunet-io/scriban/issues/54

[Tyrrrz]

I wonder if I'm not using Scriban's TextWriterOutput correctly or is it not completely "streamful".

[Tyrrrz]

I changed the code so it matches the one in the linked issue on Scriban. Hopefully, it will help with this issue, although I'm seeing that it still buffers some data regardless. Also, the newly introduced partitioning should also help with this problem.