vrtmrz
commented
5 months ago Thank you for your report! This problem has been caused by here: (new:2, recycled:2607, cached:46)
To less traffic, Self-hosted LiveSync splits the documents into multiple chunks and transfers only new things. However, checking existent from the local database is a bit heavy if there are a lot of them. And worse, this slows down according to their number.
To avoid this, Self-hosted LiveSync can cache recently used chunks. The initial value is configured in a little conservative value, but we should enlarge this in such a situation.
Please try setting Memory cache size (by total items) to 300000, and Memory cache size (by total characters) to 100 once. (Yes, this can be controlled by the count and total amount, for such a case).
I think that your environment could accept these values.
firstunicorn
Abstract
Sync takes too long/buggy with large text (.MD) files.
Expected behavior
Sync should take almost the same time as small files.
Actually happened
Reproducing procedure
Frequency: constantly
Plug-in log
Plug-in log
``` 5/1/2024, 7:01:42 PM->OneShot Sync begin... (sync) 5/1/2024, 7:01:43 PM->Replication paused 5/1/2024, 7:01:43 PM->Replication paused 5/1/2024, 7:01:43 PM->Replication completed 5/1/2024, 7:01:49 PM->Chunks saved: doc: TODO.md ,chunks: 2655 (new:2, recycled:2607, cached:46) 5/1/2024, 7:01:54 PM->STORAGE -> DB (plain) TODO.md 5/1/2024, 7:01:54 PM->OneShot Sync begin... (sync) 5/1/2024, 7:01:54 PM->OneShot Sync begin... (sync) 5/1/2024, 7:01:55 PM->OneShot Sync begin... (sync) 5/1/2024, 7:01:57 PM->Replication paused 5/1/2024, 7:01:57 PM->Replication activated 5/1/2024, 7:01:58 PM->Replication paused 5/1/2024, 7:01:58 PM->Replication completed ```Screenshots
https://i.imgur.com/zFnzCZf.png
Details and background
First of all, thank you so much for your work. This plugin is a life changer for me and I transfer all my note-taking and task management to Obsidian on all devices thanks to this plugin. But recently I encountered some bug/drawback using CouchDB. When I have a file with a lot of text (like 100 pages... it looks like just a lot of tasks / TODO list, here is an example: https://i.imgur.com/zFnzCZf.png) it becomes slow. By slow I mean really slow as hell, it takes 10-30 seconds to upload any minor changes and about 10-30 seconds more to fetch on another machine. And with changes more than minor (a few paragraphs), it can take minutes in some cases. In the beginning, with 20 pages it became unusable in LiveSync mode (because while you typing, every character takes a few seconds to sync) and I switched to "Periodic and on Events" for every 4-second interval sync mode with batch syncing turned on and it worked for a while similar to real-time sync, but after 60 pages it again becomes slow down and periodic removing CouchDB on a hoster and create it again helps, but only for short period of time and after about 100 pages it becomes totally unusable, because on top of the slow sync - it slow my laptop (Obsidian become super slow, fans on laptop goes crazy... when I turn off Self-hosted LiveSync plugin - everything backs to normal). Not to mention it slows down syncing even for other, smaller files (but not so much, but still there is a huge difference... because at the beginning when there were just a few files and they were small, it was sync changes almost immediately and now even for small files it can takes few seconds - so the difference in hundreds percents). I use AlwaysData hosting free tier to host CouchDB. I think the issue is the amount of chunks and architecture of CouchDB and/or the process of cleaning and optimizing the database because, on a fresh database, everything syncs blazingly fast. Will switching to CloudFlare R2 storage solve this issue? Is it even worth trying?