Performance issues while plugin is recounting

[leoccyao]

Description

This one was a bit of a pain to get a hypothesis for. The issue first materialized with the editor freezing for 5-10 seconds whenever creating or deleting a file. While debugging this issue by repeatedly testing creating and deleting files, lags of 2-4 seconds were witnessed with other actions as well, such as navigations (through quick switcher, Pane Relief previous/forward, and opening hover editors). Other symptoms include lags in Templater processing for file creation, or failures to open files from the quick switcher.

After some more investigation, I noticed that the issue would sporadically disappear: the first file creation after Obsidian reload would be okay, but then the deletion would lag. Profiles during lag would look like this, with a lot of file read operations occurring, but no attribution to a specific plugin:

After enabling debug mode, the issue occurring seems to be reasonably correlated with the plugin engaging in recounting of files. I have a vault of ~2k files totaling ~800k words, and these recounts often take 30-120s.

My best guess is some odd interaction occurring when a second recount is triggered during one, though this is just a guess and I am not too technically strong in JavaScript debugging.

Device

macOS Ventura, Obsidian 1.4.5, Novel Word Count 2.23.0.

[isaaclyman]

Thanks for your troubleshooting work, that's really helpful.

Previously, file creation/deletion triggered a recount of the entire vault. I wrote the plugin this way because it was easy to do and prioritized correctness, counting everything from scratch on a regular basis. However, strictly speaking it's unnecessary, and if it's causing performance issues, it's probably time for the plugin to mature and only do the necessary work on files/folders that have changed.

I've just published plugin v2.23.1. Could you try it out and let me know if you notice any more issues? Happy to reopen the ticket if it's still being slow.

[leoccyao]

After updating and enabling the plugin, it seems that the performance issues relating to creating/deleting files are gone. Thanks for the prompt response and rewrite!

However, there seem to be two issues that still persist: Upon reloading the vault, I am stuck on the "Loading workspace..." screen until a full reindex of the vault occurs, with errors in the console occurring every second as the plugin tries to find the file explorer:

Second, counts in the file explorer are 0 until the file is edited (in this example, today's note was opened and edited):

The same behavior occurs for folders as well; folders that have a file under them opened have the correct count; others have zero.

Disabling and enabling the plugin restores the correct counts, albeit after another full reindex is done.

[isaaclyman]

Seeing the same thing on one of my vaults. I've yanked the release, will look into it and see what the problem is.

[isaaclyman]

Just published v2.23.3. This version waits until the workspace is ready before analyzing files, which should help with load time in your vault. I believe I've also fixed the bug with files failing to get a count. Let me know how it works for you.

[leoccyao]

Files failing to get a count bug looks fixed! Unfortunately, still getting stuck on the "Loading workspace..." stage and seeing cycles of errors:

[isaaclyman]

Ok. So everything is working okay, aside from a steady stream of console errors?

[isaaclyman]

Note that a thread caught in a loop often won't stop until you close and restart Obsidian.

[leoccyao]

Unfortunately not; while these console errors are occurring, the app is stuck on the "Loading workspace" step of initialization, and only gets to the editor after a significant amount of time (seemingly when the count finishes)

[isaaclyman]

The problem is I don't have a vault of that size to test on, so it's all guesswork on my end. I'm assuming your vault isn't something you'd want to share publicly? Maybe I can track down a large vault online somewhere.

EDIT: Found a vault with 5,000 notes and nearly 5 million words. Stay tuned.

[isaaclyman]

Can I ask what hardware you're on? I'm on an M1 Macbook Air. After adding those 5,000 notes to my test vault, it took a minute or two for Obsidian to index them for the first time, but now if I trigger a recount from the Command Palette or restart Obsidian, it counts them all in less than 1.5 seconds.

[isaaclyman]

You know what, I think I forgot to push a commit. Just published 2.23.4. Can you update and report back? Sorry for the runaround.

[leoccyao]

Issues seem to be fixed as of the most recent version. No problem; thanks for all the prompt testing and fixes!