brianpetro / obsidian-smart-connections

Chat with your notes & see links to related content with AI embeddings. Use local models or 100+ via APIs like Claude, Gemini, ChatGPT & Llama 3
https://smartconnections.app
GNU General Public License v3.0
2.86k stars 185 forks source link

Fixed JSON stringify overload from preventing save #402

Open winne617 opened 11 months ago

winne617 commented 11 months ago

i'm currently experiencing an ongoing error: problem is that the embeddings-file doesn't get saved anymore. last time it was saved was over two weeks ago, despite my daily use of obsidian, several restarts, as well as updating the smart connections plugin.

an error message indicates an "invalid string length". image

however, no error appears in the plugin settings window. i am able to press 'manual save' and everything seems to be functioning normally. yet, the embeddings-file is not updating or saving as expected.

my vault consists of nearly 12,000 files and the embeddings-file is approximately 512 megabytes in size. image

my suspicion is that there may be some internal factor that has reached its maximum size limit? i cannot embed more notes from now? anything i can do to workaround for it?

any assistance or insight into this issue would be greatly appreciated. thank you for your time and efforts towards improving this plugin, brian.

brianpetro commented 11 months ago

@winne617

my suspicion is that there may be some internal factor that has reached its maximum size limit?

Your suspicion is correct.

In version 1, you can try the "skip sections" setting. This reduces the number embeddings to the number of files in your vault, a temporary workaround that unfortunately may impact performance.

In version 2, there will be more options, and ideally this will be avoided altogether. Due to the first-draft nature of v1, it made most sense to just "load everything everytime.". V2 is refined such that implementing improved logic, like saving to multiple files, loading them as-needed, will be easier to implement, though it hasn't been implemented yet.

In the meantime, you could also use the exclusion settings in v1 to exclude any large folders that may not be needed in the short-term.

Hope that helps,

🌴

winne617 commented 11 months ago

ok, thanks for clarification. excluded many folders already. sections i use heavily and need that for my workflow. so ultimately i hope v2 is gonna make it. have a nice year btw 🥳