KevinBatdorf / obsidian-raycast

Raycast extension with commands for the note taking app Obsidian.
MIT License
108 stars 19 forks source link

JS Heap Out of Memory Issue When Searching for Notes or Creating New Notes #93

Open DerDoktorFaust opened 3 months ago

DerDoktorFaust commented 3 months ago

Hello!

I am running into the following error while trying to use the extension:

Error: Worker terminated due to reaching memory limit: JS heap out of memory

Worker..onexit:worker:229:20

This occurred when I tried to search through my notes for a particular tag.

I also get the exact same error if I try to create a new note.

My paths are all correct. I can open my vault directly from the extension. But actually doing anything in the vault does not work due to the JS heap out of memory error.

I searched around on GitHub for others with this issue. I saw that some thought it might be that the extension is trying to load the entire database into memory so it can perform the search--I have no idea if that is the issue, but thought I might put it out there.

Thank you!

KevinBatdorf commented 3 months ago

Do you have a lot of notes? If so then that's probably the case. The extension is built around loading everything into memory for searching, etc. and it would require rewriting it to either use the obsidian sdk or even adding a custom index.

I'd have to look at the code more closely, but running out of memory when creating a note, if it's because of loading the entire db into memory, that could probably be optimized.

DerDoktorFaust commented 3 months ago

Thank you for the response. I have about 2500 notes. Some have several thousand words, most are 200-1000 words.

How small is the memory allocation? Is it artificially limited by Raycast or something else? Because I have far more than enough RAM to load the entire database.

I do have one or two PDFs and 20 to 30 images, does it load those too? They are small, but if the JS heap is low, that might eat it up.

Edit:

I looked at the cache source file and it says the cache is set for 500MB. I can confirm the entirety of my database of notes, images, and PDFs is around 50-60 MB. Or am I looking at the wrong source code for this issue?

KevinBatdorf commented 3 months ago

Is it artificially limited by Raycast or something else? Because I have far more than enough RAM to load the entire database.

I think Raycast sets a runtime memory limit of 100mb max per extension. Each runs in it's own V8 (the JS engine) process so they limit the resources.

Bunch of reports on other extensions about it. https://github.com/search?q=repo%3Araycast%2Fextensions+memory+limit&type=issues

I'll look into how much effort it would be to at least port some of the functionality over to the obsidian sdk, but there could be other blockers I'm not thinking of like permissions or something else. The sdk could also just be loading the entire store into memory too as well.

DerDoktorFaust commented 3 months ago

Ok, that makes sense! Thank you very much!

Hopefully there is some easy fix around the 100MB limit from Raycast. I don't think I have a particularly large Obsidian database, so I have to think a lot of people are running into this issue.