Keeps re-creating my embeddings.

[bbecausereasonss]

I'm using Obsidian on Desktop/Mac and Sync with Dropbox. My embeddings keep getting re-created, seemingly every day sometimes fully. Not sure why. This never used to happen before. Also when I click 'save' now nothing happens where previous versions used to save an embedding file.

[brianpetro]

@bbecausereasonss are you talking about the "save" by the API key? There isn't currently a method for manually saving. However, checking the developer console logs should let us know whether the file is being saved. It should say something like "Saved in XXXXms".

[bbecausereasonss]

No I mean the save .smart-connections data

[brianpetro]

That button simply saves the setting. It's a manual save for the setting because if you change the folder, it needs to rename it, which you don't want to trigger automatically.

[DantesHub]

Hey @brianpetro I'm having the same problem, this prompts keeps appearing everytime i reopen obsidian.

[DantesHub]

[robwheatley]

I have the same problem. My vault is saved to iCloud rather than dropbox and I assume it has something to do with the file dates. I'm wondering if iCloud/DropBox messes with the dates on files so Smart Connections things that the embeddings are out of date. I came here to see if that was the case. Not found anything yet, but perhaps this will give people a clue?

[sh4d0wl3ss]

Same issue and I am using Obisidan Sync but also generate daily journal entries and have about 8k notes in the vault. I'm spending much more time with degraded Obsidian performance while it embeds smart notes on the same folder every day than I actually use the smart connections features. Juice isn't worth the squeeze. Didn't do this before the latest Obsidian update.

[brianpetro]

@robwheatley @DantesHub @sh4d0wl3ss

Keep an eye on the dev console during the embedding process. If there's an error with saving, screenshot and I can fix that.

If everything saves correctly, then something else is clearing the embeddings. Check the dev console immediately after startup to check for loading issues.

Otherwise, it might be something that's specific to your setup. So also include any third-party syncing that you're using, like how @robwheatley mentioned he is using iCloud.

Thanks for your help in solving this issue, 🌴

[brianpetro]

Also, which operating system you're using will help narrow down the issue.

[robwheatley]

Hi @brianpetro I'm new to Obsidian and didn't realise that there was a dev console. I've just taken a look and have spotted a few errors being thrown out by smart connections. Not sure if this helps you, but I can provide more info on request...

I'm on oSX 14.3.1 and Obsidian 1.5.8 and smart connections 1.0.128. Also, as mentioned, I have my vault in iCloud.

As a test, I moved my vault out of iCloud and onto my regular drive. When I did that, the dev log did look a little different - I more entries for "Embedded X inputs..." but I still got the final undefined 'last_history' at the end of the log like below. Also, on restart of Obsidian, all my notes required embedding again - so it doesn't look like an iCloud specific problem.

Here is the log when running on iCloud...

Edit: Please only upload screenshots of logs

[brianpetro]

@robwheatley if you could screenshot the console, that would be much easier for me to look through.

Also, toggle this on for better logs 🌴

[robwheatley]

@brianpetro here is the 1st screenshot - taken after start-up with the debugging on. The 2nd screenshot is from after I clicked the 'create embeddings' button up to the point it finished

Shot 1

Shot 2

[brianpetro]

Does anyone that this has happened to remember hitting the pause button prior to losing the embeddings?

It seems like under some conditions, like pausing/restarting, multiple embedding processes could be executed at once. This is visualized by the denominator in the progress notification changing between multiple values.

So far this has only happened once for me during development, so it'll require more testing.

There could be some other situations where this happens. I'm continuing to investigate.

[robwheatley]

Does anyone that this has happened to remember hitting the pause button prior to losing the embeddings?

It seems like under some conditions, like pausing/restarting, multiple embedding processes could be executed at once. This is visualized by the denominator in the progress notification changing between multiple values.

So far this has only happened once for me during development, so it'll require more testing.

There could be some other situations where this happens. I'm continuing to investigate.

I'm 99% sure I didn't pause. OK, 98%

[bbecausereasonss]

Does anyone that this has happened to remember hitting the pause button prior to losing the embeddings?

It seems like under some conditions, like pausing/restarting, multiple embedding processes could be executed at once. This is visualized by the denominator in the progress notification changing between multiple values.

So far this has only happened once for me during development, so it'll require more testing.

There could be some other situations where this happens. I'm continuing to investigate.

I hit the pause button once, but that was weeks ago and it's been happening ever since.

[brianpetro]

@robwheatley @bbecausereasonss thanks for letting me know!

🌴

[brianpetro]

Thinking out loud here:

Something else that was recently changed was replacing a file hash (b/c incompatible with mobile) with checking both the files size and last change time.

In theory, this should not cause a noticeable difference because even if the time is modified by some other process, the file size should stay the same.

But, in practice, maybe the file size is also being altered even without changing the note.

I'm going to need to come up with some sort of test for this.

🌴

[robwheatley]

I've been doing some messing about with a new Vault saved directly onto my hard drive. Clean install, with only the Smart Connections plugin installed. And I've been adding notes to that to see what it does....

I think what's happening is that when the embeddings.ajson file is loaded on start-up, the JSAON parser doesn't like something and reports the error below. That results in the embedding.ajson file being deleted, so you have to start the embedding again.

SyntaxError: Bad control character in string literal in JSON at position 3054377 (line 1 column 3054378)
    at JSON.parse (<anonymous>)
    at ObsAJSON2.load (plugin:smart-connections:105:31)
    at async SmartNotes.load (plugin:smart-connections:2054:9)
    at async eval (plugin:smart-connections:2767:92)
    at async Promise.all (index 0)
    at async ScBrain2.init (plugin:smart-connections:2767:9)
    at async ScBrain2.reload (plugin:smart-connections:2750:9)
    at async SmartView2.load_brain (plugin:smart-connections:4293:9)
    at async SmartView2.initialize (plugin:smart-connections:4283:9)

I don't know what the bad character is - I can't see anything obviously wrong with the file (I can do more testing later) and I don't know what put it here in the first place. I wonder if I could hack something to prevent the file from being deleted when it discovers the error to see what's going on....?

[brianpetro]

@robwheatley good catch!

That error doesn't specifically delete the file, but the embeddings fail to load and then the reprocessing overwrites the existing file.

Solving the source of the issue: I'm thinking it's this line https://github.com/brianpetro/obsidian-smart-connections/blob/077bf4f5e6d591bc90a7eeece7c7966de57459bc/src/smart_entities.js#L165

Specifically, "${this.key.replace(/"/g, '\\"')}": should be ${JSON.stringify(this.key)} to handle any control characters.

I'll get this change shipped in the next update, today if I can fit it in.

Another thing that can help situations like this: saving the file so that records (or batches of records) are separated by newlines. This way the erroneous record/batch can be thrown out while preserving the rest. This would likely have a negative impact on start-up performance, but would still probably be worth it to prevent this embedding-rewrite headache.

🌴

[robwheatley]

@brianpetro It's great that you are looking into this. I've just spent the last few hours seeing if I could add anymore info. I went down a bit of a rabbithole TBH!

From my clean install, I started to add notes in from my 'real' vault. I wondered if a particular note was causing the issue. After lots of messing about I thought I found something. When I added a specific note, I started to get errors. But it turned out to be nothing special about that note. If I just added 'one more note' of any sort, I would cause the issue. Basically, I got in the situation where I had 236 notes, but adding a 237th would make things fail.

I then started to look other things, because adding the 237th note doesn't 100% reproduce the problem. So I then started to add more content to notes when I just had 236. I'm not convinced that this actually got me anywhere though!!

I did run into a few odd things along the way though. For example, when I added a new note, I got the alert to say that it was being embedded, but the alert never went away, even though I could see in the console that some sort of embedding had been done. Sometimes I saw a time-out on this single file (dunno why, I was using a super simple embedding and I'm on a speedy machine). Also, on start-up, I sometimes get asked if I want to re-embed all my notes, even though I can see there is a valid embeddings file and there has been no parsing errors. Quit and restart sorts that out (on next run, I'm not asked to re-embed). I'm not sure how you are keeping track of what's been embedded or not. Maybe the embeddings file itself, and these issues were being caused by file-on-disk mismatches. No idea, and I realise these ramblings won't help!

I'm super-keen to get this plugin working though. I've only just moved to Obsidian, and although I'm putting some structure in place for new notes, the old ones I have imported are a mess, so this would be really useful!

[brianpetro]

@robwheatley thanks for sharing all that! Your rabbit hole can be my gold mine. It's not very often (considering how many people have downloaded Smart Connections) that I get such detailed feedback 😊

You definitely managed to point out some curiosities.

Separating meta data from embeddings files is something I've played around with in the past, and could be a way to thwart some of these issues.

If you ever have a note that seems to cause an issue, but you can't figure out why, please do share the note with me. If you need a private channel to do so, I can accommodate. But being able to see some of these issues myself can be invaluable to the debugging process.

There is still a lot of legacy code in v2.0, but I'm continuing to modularize the processes, enabling useful test processes, so the stability will only improve (though things tend to get worse before they get better...). More importantly, these design decisions should also allow contributing by community members long into the future.

PS- all these GitHub issues end up in my personal obsidian vault, and Smart Connections enables me to resurface them at the right times. So any notes you make on your experience, much like what you just shared, will be useful even if they aren't specifically addressed right away.

Thanks for your help in making Smart Connections better! 🌴

[brianpetro]

@robwheatley @bbecausereasonss @DantesHub @sh4d0wl3ss latest update v2.0.129 implements https://github.com/brianpetro/obsidian-smart-connections/issues/496#issuecomment-1983416365

[robwheatley]

@brianpetro No joy I'm afraid. I updated to the latest version, scrubbed everything and started embedding from scratch.

My local vault created the embeddings file, but on restart wants me to re-do them all again (even though there is a valid 2Meg file there, that doesn't get overwritten). I can get more from the logs on this later.

My iCloud vault created the embeddings (took at lot longer as I have 4x more notes in this one), it said that it saved the file in the console, but the saved file was zero bytes, and obviously I get asked to re-do them on restart.

Must be something else causing the issue. I don't think I will have much time to play tonight, but will over the weekend if you don't work it out before then....

[brianpetro]

@robwheatley bummer, but thanks for letting me know.

When you get a chance, let me know if you're still seeing the same error or if it's something new.

🌴

[brianpetro]

@robwheatley @bbecausereasonss @DantesHub @sh4d0wl3ss released another update v2.0.130 to try to address this. In short, a temporary file is created on save to prevent overwriting the existing file 🌴

[robwheatley]

@brianpetro I tried .30 and it looked liked it worked, but saw a couple of funny things. I wasn't paying much attention as I was waiting in the car for my daughter. Just got back in the house and seen that you have released a .31 version.

Happy to report that I've hit no issues so far (local or iCloud). To flex it a bit, I've just switched to a beefier model to see if a larger file size causes any issues. It's chugging away as I type and I can see that you are saving the file after a few notes have been processed and recording the file sizes as you go. At least I can see the size increasing, so that's encouraging!

YES! IT WORKED - Nice one!! I've quit and restarted several times and not been asked to re-embed everything again

[robwheatley]

@brianpetro Spoke too soon. It creates the notes embeddings fine, but it's failing to create block embeddings at the moment. I'm getting this error. Just thought I should let you know.

plugin:smart-connections:2520 Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'last_history')
    at SmartBlock.init (plugin:smart-connections:2520:37)
    at SmartBlocks.create_or_update (plugin:smart-connections:763:14)
    at eval (plugin:smart-connections:2460:42)
    at Array.forEach (<anonymous>)
    at eval (plugin:smart-connections:2460:18)
    at async Promise.all (index 284)
    at async SmartBlocks.import (plugin:smart-connections:2457:9)
init @ plugin:smart-connections:2520
create_or_update @ plugin:smart-connections:763
eval @ plugin:smart-connections:2460
eval @ plugin:smart-connections:2460
await in eval (async)
ensure_embeddings @ plugin:smart-connections:2277

[brianpetro]

@robwheatley, thanks for the update! Seems like we're at least making some progress 😊

Please toggle on this option:

It makes the logs provide useful line numbers (the other ones are based on a compiled file).

I just made an update (v2.0.134) to log the blocks causing the above error. This update should give us a better idea of what's going on with that error by letting us see the blocks that are causing it.

Thanks for your help 🌴

[robwheatley]

I've just deleted the previous comment saying that all is well. My embeddings file just got overwritten with an empty file after a quit and restart. I wasn't paying attention as I was doing other things at the time. I will keep an eye on things and add more info when I can...Sorry to be giving you bad news on a weekend..

[brianpetro]

@robwheatley you jinxed it! Lol.

In the latest version, I added logic so that, when new embeddings are being saved, the disk writes happen in a new temporary file. That new temporary file should only replace the existing "working" file if it is at least 50% of the size of the "working" file. So it's weird that you would end up with a completely empty file.

A few things to check:

1) Make sure you are on the latest version so we can rule that out. The latest version is v2.0.134.

2) If you're a supporter, are you also running the Smart Connect software when this happens?

3) Are there any third-party syncing processes that might be overwriting the file? If so, can the .smart-connections folder be excluded from that syncing?

4) Screenshots of anything that might be of interest are always helpful.

Thanks for the update 🌴

[robwheatley]

Sorry - I've nothing to give you. I've just spent the last hour trying to reproduce the problem (any problem in fact) but I can't, so I've nothing to share. Everything is working fine. I guess we should be happy about that, but I'm a pessimistic kinda guy and feel there is something lurking under the hood! In the mean time, shall we just cross our fingers?

[brianpetro]

@robwheatley sometimes all you can do is pray to the tech voodoo gods 🤷

[bbecausereasonss]

Again, just got off my PC (where all my embeddings were created/working) opened my Mac (where it did a sync with dropbox) and BAM. Trying to re-create my entire embedding library.

[avataraustin]

I may have a similar issue. I had been trying to use two laptops to share a vault via icloud. I was having issues with it wanting to re-embed when I try to use one of the other machines. Today I got back and went to use my main laptop after a vacation. I thought I might be ok because it only popped up a message asking if I wanted to embed 58 smart connections after I updated to 2.0.135. I clicked to go ahead, but then I got a popup saying it would also delete 15,891 (99%) Block-level embeddings..... I clicked cancel.

I also opened the dev console I copied everything it said and I will now paste that information below in case it may help:

---

Edit: Please only upload screenshots of logs

---

[brianpetro]

@bbecausereasonss @avataraustin, I'd be interested in seeing a screenshot of the logs immediately after this happens, prior to re-embedding. If it happens again, click cancel on the prompt and screenshot what's in the logs.

If you recently updated, there may have been a change that triggered the re-embedding. @avataraustin depending on your previous version, this could have been what happened. If that's the case, then it shouldn't happen again.

🌴

[brianpetro]

Debug note:

• [ ] DO add debug_changed setting (toggle) to log size and mtime of "changed" entities (both stored and current) for review

[bbecausereasonss]

Here's my log on PC (Win11) after I redid the embeddings.

Edit: Please only upload screenshots of logs

[brianpetro]

@bbecausereasonss a screenshot of the logs after the issue occurs, but before the embedding process starts, will let me see whether there might be a problem loading the embedding 🌴

[Hopsakee]

I'm having the same issue. These are screenshots before I restart the actual embedding. Just before this happened I nicely embedded just 12 files I had created. But some time later it wanted to embed all notes again. I use Obsidian Sync.

I'm not completely sure, but I think the first screenshot shows the initial embeddings of 12 files. But I already have a lot of errors, so don't know for sure.

Hope this helps.

Sincerely,

Jelle

p.s I tried reinstalling. I use local embeding models, not OpenAI.

[brianpetro]

@Hopsakee that is helpful, thank you!

For the future, if you could turn on the option seen in https://github.com/brianpetro/obsidian-smart-connections/issues/496#issuecomment-1984848306 that would make the logs even more helpful.

Can you confirm that you're on v2.0.135?

Thanks for your help in solving this 🌴

[bbecausereasonss]

I'm on the beta.

On Wed, Mar 13, 2024 at 7:02 PM WFH Brian @.***> wrote:

@Hopsakee https://github.com/Hopsakee that is helpful, thank you!

For the future, if you could turn on the option seen in #496 (comment) https://github.com/brianpetro/obsidian-smart-connections/issues/496#issuecomment-1984848306 that would make the logs even more helpful.

Can you confirm that you're on v2.0.135?

Thanks for your help in solving this 🌴

— Reply to this email directly, view it on GitHub https://github.com/brianpetro/obsidian-smart-connections/issues/496#issuecomment-1996051310, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE2TIUCWMMLPPYNE5ETQT33YYDLIPAVCNFSM6AAAAABEAZAMH6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJWGA2TCMZRGA . You are receiving this because you were mentioned.Message ID: @.***>

--

Vladimir Druts Tech & DTC Founder 🌄 Helping you focus and thrive in a digital world DoneSquad http://donesquad.io/ 🎯 Accountability Program FocusMe http://focusme.com/ 🎯 Attention Optimization App FRWRD Skincare https://www.frwrdskincare.com/ 🍄 Mushroom Beauty & Wellness

🎙️ Podcasts & Articles http://linktr.ee/vdruts 🏷️ #ADHDFocused #HighPerformance #TechForGood #WellnessAdvocate

SCHEDULE A CALL https://tidycal.com/vd

[Hopsakee]

You're welcome, although reporting this and sharing my log is also in my own interest of course. I love this plugin.

I am on v2.0.13

I also tried to keep the '.smart-connections' folder from syncing with Obsidian sync. But I don't think that will work, because I don't think it works this way with hidden folder.

But I thought I should mention this, in case my debug log shows funny stuff.

Here is a screenshot with all my plugins turned off, except for smart-connections. And with Debug startup time on. Hope this helps.

[brianpetro]

@Hopsakee thanks for the follow-up, that helps a lot.

1) have you completed the (re-)embedding process for all the files since this started happening? If not, you can try the "force refresh" in the settings. This would clear up any previous issues that have since been fixed.

2) to avoid re-embedding, you could try find/replace, searching for }{ and replacing with },{. But this would only help if the issue was caused by the bug from a few versions ago that was since fixed.

Let me know if either of those work. If not, I'll have to keep exploring what might be causing this.

Thanks for your help, and happy to hear that you love Smart Connections 😊

🌴

[bbecausereasonss]

Just had the issue come up again, was working on macbook, redid a ton of embeddings. Got on my PC and it's asking me to start the embedding process. I have not clicked to start it, here's the dump.

Obsidian Developer Console plugin:obsidian-icon-folder:5427 loading iconize plugin:obsidian-mind-map:32574 Loading Mind Map plugin plugin:obsidian-tasks-plugin:149 [2024-03-14-12:13:08.945][info][tasks] loading plugin "Tasks" v6.1.2 plugin:obsidian-minimal-settings:80 Folding is on plugin:obsidian-minimal-settings:94 Line numbers are off plugin:obsidian-minimal-settings:100 Readable line length is on plugin:simple-embeds:29615 Loading Simple Embeds v1.15.0 plugin:obsidian-sortable:180 Sortable: loading plugin... plugin:obsidian-sortable:183 Sortable: loaded plugin. plugin:highlightr-plugin:3508 Highlightr v1.2.2 loaded plugin:table-editor-obsidian:28854 loading markdown-table-editor plugin plugin:webpage-html-export:74719 Loading webpage-html-export plugin plugin:file-explorer-note-count:933 loading FileExplorerNoteCount plugin:editing-toolbar:9 cMenuToolbar v2.4.0 loaded plugin:obsidian-auto-link-title:404 loading obsidian-auto-link-title plugin:obsidian-paste-image-rename:465 Plugin loading: obsidian-paste-image-rename 1.6.1 BUILD_ENV=production plugin:recent-files-obsidian:234 Recent Files: Loading plugin v1.3.10 plugin:recent-files-obsidian:272 Recent Files: maxLength is not set, using default (50) plugin:arrows:52 Loading Obsidian Arrows... plugin:arrows:62 Loaded Obsidian Arrows. plugin:link-favicon:9 enabling plugin: link favicons plugin:obsidian-shortcuts-for-starred-files:96 loading Hotkeys for Bookmarks 2plugin:lovely-mindmap:495 setting plugin:obsidian42-brat:2300 loading BRAT plugin:obsidian-focus-mode:152 Loading Focus Mode plugin ... plugin:obsidian-markdown-formatting-assistant-plugin:2364 Object plugin:obsidian-markdown-formatting-assistant-plugin:4815 loading obsidian-markdown-formatting-assistant-plugin plugin:copilot:80585 New LLM chain created. plugin:copilot:91572 Setting model to GPT-4: gpt-4 plugin:pdf-to-markdown-plugin:62114 Found existing settings file plugin:smart-connections:7704 Loading Smart Connections v2... plugin:callout-manager:1620 Unable to fetch Obsidian stylesheet. Error: [obsidian-extra]: Obsidian 1.5.8 has not been tested with this function at eval (plugin:callout-manager:154:13) at Generator.next () at eval (plugin:callout-manager:147:67) at new Promise () at __awaiter (plugin:callout-manager:129:10) at fetchObsidianStyles (plugin:callout-manager:152:10) at StylesheetWatcher.checkForChangesObsidian (plugin:callout-manager:1612:35) at StylesheetWatcher.checkForChanges (plugin:callout-manager:1594:28) at StylesheetWatcher.watch (plugin:callout-manager:1531:10) at eval (plugin:callout-manager:2983:37) at t. (app.js:1:2003310) at app.js:1:237544 at Object.next (app.js:1:237649) at a (app.js:1:236367) checkForChangesObsidian @ plugin:callout-manager:1620 plugin:obsidian-excalidraw-plugin:95 Initialized Excalidraw Image Cache plugin:smart-connections:5337 get_conversation_ids_and_file_types plugin:smart-connections:5339 Object plugin:smart-connections:7005 SmartConversationMD plugin:smart-connections:6664 init SmartChatRenderer plugin:smart-connections:6665 <div class=​"workspace-leaf-content" data-type=​"smart-connections-chat-view">​…​

[brianpetro]

@bbecausereasonss are you syncing the .smart-connections folder?

Also, I'd appreciate if you could provide screenshots in the future instead of copy&pasting.

🌴

[bbecausereasonss]

@bbecausereasonss are you syncing the .smart-connections folder?

Also, I'd appreciate if you could provide screenshots in the future instead of copy&pasting.

🌴

Sorry will do. Yes, I believe the .smart-connections folder is being synced. I'm syncing the entire /obsidian vault folder.

[brianpetro]

@bbecausereasonss Thanks, it makes things easier for me to read.

I think the syncing is causing the issue. Mac and windows may report different file sizes for the same file. And it's even possible on the same OS with different underlying architectures.

If that's the case, if you disable syncing that folder, then the re-embedding should stop.

---

@Hopsakee by default, Obsidian sync should not sync hidden folders.

🌴

[bbecausereasonss]

@bbecausereasonss Thanks, it makes things easier for me to read.

I think the syncing is causing the issue. Mac and windows may report different file sizes for the same file. And it's even possible on the same OS with different underlying architectures.

If that's the case, if you disable syncing that folder, then the re-embedding should stop.

@Hopsakee by default, Obsidian sync should not sync hidden folders.

🌴

One issue with Dropbox is that you cannot selectively sync a folder. As it will rename the folder (conflict xyz).

The only way to remedy this is to be able to place the smart connections folder outside of the vault folder.

Is this possible? Right now it asks where to place it, but is that a relative path within the Obsidian vault only?

[brianpetro]

@bbecausereasonss

Is this possible? Right now it asks where to place it, but is that a relative path within the Obsidian vault only?

Unfortunately, I don't think so because the Obsidian file-system adapter being relative to the vault path.

Depending on how Dropbox handles symlinks, maybe you could create a symlink for .smart-connections folder that exists outside the vault.

If this continues to be a problem, maybe in the future I could add a toggle for using hashing as the change-detector again, but I removed it because it's incompatible with the mobile app, which I'm trying to support.

🌴

[bbecausereasonss]

@bbecausereasonss

Is this possible? Right now it asks where to place it, but is that a relative path within the Obsidian vault only?

Unfortunately, I don't think so because the Obsidian file-system adapter being relative to the vault path.

Depending on how Dropbox handles symlinks, maybe you could create a symlink for .smart-connections folder that exists outside the vault.

If this continues to be a problem, maybe in the future I could add a toggle for using hashing as the change-detector again, but I removed it because it's incompatible with the mobile app, which I'm trying to support.

🌴

I'll try a symlink, but doubtful it will change much. Yes, when you used the other method it never presented an issue.