search

logseq / logseq

A privacy-first, open-source platform for knowledge management and collaboration. Download link: http://github.com/logseq/logseq/releases. roadmap: http://trello.com/b/8txSM12G/roadmap
https://logseq.com
GNU Affero General Public License v3.0
30.07k stars 1.76k forks source link

Issue with Large Logseq Graphs Exceeding 300MB Leading to Startup Failures #11236

Open Lluvia95 opened 1 month ago

Lluvia95 commented 1 month ago

Search first

What Happened?

Hello everyone,

I’m reaching out to the community for assistance with an issue I’ve encountered using Logseq with a significantly large graph. My current graph has grown to include over 30,000 files and continues to expand. However, I’ve observed a consistent issue where the application fails to launch whenever the graph’s size surpasses 300MB.

Has anyone experienced similar issues with large graphs? Any insights, workarounds, or solutions would be greatly appreciated.

Reproduce the Bug

Make a graph has a lots of files and links, till the graph’s size surpasses 300MB.

Expected Behavior

No response

Screenshots

No response

Desktop or Mobile Platform Information

No response

Additional Context

No response

Are you willing to submit a PR? If you know how to fix the bug.

andelf commented 1 month ago

Currently, Logseq loads all data into memory as a datascript DB. So this is the root cause for performance issues in large graphs. May I know what's in your graph? How many pages and journals and how many assets files? And what's the size of the largest file?

Lluvia95 commented 4 weeks ago

Thank you for explaining the performance issues with large graphs in Logseq due to how data is loaded into memory. Indeed, I do have a quite extensive graph with 69,581 pages, and tens of thousands of asset files. In terms of solutions, would upgrading my computer's RAM significantly mitigate these performance issues?

Lluvia95 commented 3 weeks ago
image

Every time I open Logseq with a graph larger than 300Mb, after a while, the console ends up looking like this.

Lluvia95 commented 3 weeks ago

I've now backed up a graph that's 299Mb in size, and every time I restart Logseq, I have to endure a lengthy wait before it opens.

Lluvia95 commented 1 week ago

Thank you for explaining the performance issues with large graphs in Logseq due to how data is loaded into memory. I have a quite extensive graph, consisting of 69,581 pages and tens of thousands of asset files, which likely contributes to the slowdowns I've been experiencing.

Interestingly, I noticed that when I open Logseq with a backup of my 299MB graph, the application completes the reindexing and then exits immediately. After this process, the graph's size increases to 310MB, leading me to suspect that the issue may not solely be related to my system's memory. It could be a software issue, possibly with the program hitting a critical limit during initialization, causing an overflow or crash.

Would you have any recommendations on how to address this behavior or strategies to mitigate it?

Thank you for your guidance!