search

Tyrrrz / DiscordChatExporter

Exports Discord chat logs to a file
MIT License
7.5k stars 683 forks source link

OutOfMemory bug when rendering a large channel #125

Closed Torobro closed 4 years ago

Torobro commented 5 years ago

I have a PC with 16GB, when DiscordChatExpoter rendered the large channel, it not responding then it show OutOfMemory error,when i click "ok", it crashed

Tyrrrz commented 5 years ago

Which version are you using?

Tyrrrz commented 5 years ago

Just saw the title. It's possible the channel is too large to download all at once. Have you tried date ranges?

Torobro commented 5 years ago

No I have a alt rn so don't doubt the alt

Tyrrrz commented 5 years ago

Try using date ranges or partitioning

Haukusan commented 5 years ago

Still crash

riptl commented 5 years ago

Imo, the messages should be streamed to the output. (For CSV at least)

lakecityransom commented 5 years ago

I can confirm this. 250k+ msg channel over 4 years.

Sometimes says this instead. Not sure if related or a symptom: Error occurred HtmlShared.html(81,73): error: Unexpected exception when calling FormatMarkdown

I tried to narrow scope of download to see what was causing this, but kept running into this bug report's OOM msg. When I narrowed to months I got everything, but it was very painful.

Tyrrrz commented 5 years ago

Just to confirm, does it help if you use partitioning?

lakecityransom commented 5 years ago

I was kind of fixated on getting 1 file or exact time frames I should have just done that. Well, I tried it twice today. Once at 100k partition and another at 50k partition and it failed both times with out of memory. Other channels in excess of 100k msgs have pulled with no partition. Probably irrelevant but just FYI I have 32GB system memory.

Error occured

System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.

at System.Text.StringBuilder.set_Length(Int32 value)

at Scriban.Template.d__5.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at System.Threading.Tasks.ValueTask`1.get_Result()

at Scriban.Functions.IncludeFunction.d__2.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at System.Threading.Tasks.ValueTask`1.get_Result()

at Scriban.Syntax.ScriptFunctionCall.d__0.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at Scriban.Syntax.ScriptFunctionCall.d__1.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at Scriban.TemplateContext.d__2.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at Scriban.TemplateContext.d__0.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at Scriban.TemplateContext.d__1.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at Scriban.Syntax.ScriptExpressionStatement.d__0.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at Scriban.TemplateContext.d__2.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at Scriban.TemplateContext.d__0.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at Scriban.TemplateContext.d__1.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at Scriban.Syntax.ScriptBlockStatement.d__0.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at Scriban.TemplateContext.d__2.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at Scriban.TemplateContext.d__0.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at Scriban.TemplateContext.d__1.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at Scriban.Syntax.ScriptPage.d__0.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at Scriban.TemplateContext.d__2.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at Scriban.TemplateContext.d__0.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at Scriban.TemplateContext.d__1.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at DiscordChatExporter.Core.Rendering.HtmlChatLogRenderer.<>c__DisplayClass10_0.<b__0>d.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at DiscordChatExporter.Core.Rendering.HtmlChatLogRenderer.d__10.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at DiscordChatExporter.Core.Services.ExportService.d__3.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at DiscordChatExporter.Core.Services.ExportService.d__4.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)

at DiscordChatExporter.Gui.ViewModels.RootViewModel.d__49.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.CompilerServices.AsyncMethodBuilderCore.<>c.b__6_0(Object state)

at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)

at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)

OK

lakecityransom commented 5 years ago

My apologies I just noticed today that despite kicking this error out, a 25k partition did output. I just never checked. I happened to look today and saw it.

Catzzye commented 5 years ago

I am having the same problem for the channel with tons of messages over a few years. I am not sure how this could be fixable for the developer, we as users should just go bit by bit using dates.

Tyrrrz commented 4 years ago

FYI this is planned as one of the top priority fixes once I'm able to work on DCE again

TheRenegadist commented 4 years ago

Thank you for looking into fixing it instead of telling people to just use dates. Looking forward to the update and thanks for taking the time to work on it.

TheRenegadist commented 4 years ago

Is there any ETA on the fix/update?

Tyrrrz commented 4 years ago

No ETA. If someone has an idea how to stream more-or-less structured HTML without manually writing every line, I would be happy to hear about it.

zetha1 commented 4 years ago

The problem is, in my opinion, the HTML structure. It has not been designed for such thing. Even if you find a way to resolve this Outofmemory exception, I'm not sure you will be able to open the result HTML file... It would take forever to open! A100MB html file already takes more than 15min to open on my very decent computer (core i7, 128GB RAM, SSD etc...).

At your place, I would take a different approach: 1) Save the content of a channel in a structured database instead, like SQLite. 2) Format and export the content of the channel in a form that can handle huge amount of information. (PDF maybe? Or a JSON + HTML, with a function to load the data step by step, and not all the file at once)

With a database, you will have the flexibility to evolve your tool more easily in the future.