Making smart connections...

[billpetro]

Brian,

After installing via BRAT and adding my OpenAI token several days ago, I'm still getting:

Making smart connections...

I have only 4K files.

Is there a log I could check?

Thanks,

Bill Petro billpetro.com

[aikruger]

Same problem here

[brianpetro]

Thanks for the report! And yes, you can check the "console log," you can learn how to access it here https://forum.obsidian.md/t/how-to-access-the-console/16703

This issue shows an example of what it looks like https://github.com/brianpetro/obsidian-smart-connections/issues/12

If you could also let me know which patch version you're using, that could be helpful in helping me figure this out.

The patch version is the last number in the green circle in this screenshot:

Thanks @billpetro & @aikruger

[billpetro]

Brian,

Smart Connections:

Version: 1.0.26

Obsidian:

On macOS, version 1.1.9

On iOS/iPadOS, version 1.4.1 (83) API v1.0.3...

But: I installed Smart Connections with BRAT (Beta Reviewers Auto-update Tool) and it throws this error on my mobile devices:

[Obsidiann version of the app needs to be 1.1.0, but this installation of Obsidian is 1.0.3]

[billpetro]

Brian,

I uninstalled the plugin, and re-installed it fresh. I'm now on Version: 1.0.30

[brianpetro]

Thanks for following-up @billpetro

And great, 1.0.30 should work as expected on the desktop.

Mobile compatibility is in the works, but it isn't available right now. The app manifest is currently set to isDesktopOnly, this tells Obsidian that the plugin should not be available on mobile.

Are you having any issues with the desktop version?

[billpetro]

No joy. Still "Making smart connections..."

Desktop: v1.1.11 (Installer: v1.1.9) Smart Connections: v1.0.35

[brianpetro]

Can you check the "console log" for me to see if there are errors? You can learn how to access it here https://forum.obsidian.md/t/how-to-access-the-console/16703

This issue shows an example of what an error might look like https://github.com/brianpetro/obsidian-smart-connections/issues/12

Then I can better debug from there.

[Eleksar387]

Version 1.0.35

[brianpetro]

Thanks @Eleksar387

The 429 suggests that your notes aren't yet fully embedded, so the OpenAI API returns 429 for too many requests too quickly. This isn't concerning unless it happens continuously. And it will happen more at the beginning for larger vaults.

The following error was a bug that occurred because only partial data was stored after requesting the embeddings from OpenAI. The vectors that are used in the calculation failed to be stored. So I added a check to make sure they are present before continuing.

1.0.36 introduces some additional checks to prevent the error from occurring. Let me know if this helps, you get more errors, or if you continue to get 429 errors.

[mattgrieser]

I'm using Smart Connections 1.0.36 on Obsidian 1.1.11 on macOS and I'm getting the same "Making smart connections …" message.

[brianpetro]

Hey Matt,

Can you check the "console log" for me to see if there are errors? You can learn how to access it here https://forum.obsidian.md/t/how-to-access-the-console/16703

You can see the above comments for what an error might look like.

Thanks for your help.

[mattgrieser]

Brian, I'm not seeing any errors in the console. I have a .smart-connections folder with an embeddings file, but nothing else.

[brianpetro]

That's helpful. I'm going to see what I can do to reproduce the issue. There's also a couple toggle options in the settings that send more info to the console log, which may have some hints into what's happening. If you turn those on and see something let me know. Thanks

[mattgrieser]

I've got those on, but I'm still not seeing any activity in the console log. Can I filter the console to just show this plugin?

[brianpetro]

There should be a "clear console" (circle with slash) button at the top of the console to clear the logs.

Then go-to the community plugins section and deactivate then active the plugin.

After that, run the "Find Smart Connections" command using the Obsidian command palette. You'll need to have a note "active" (click on and in view) for this command to appear.

That should cause an error to appear in the console.

If there are a lot of other things in the console during this process, that may also be interesting to know in case there is some sort of interference happening.

Thanks for your help with this!

[mattgrieser]

I was getting a ton of log info from the kindle highlights plugin, so I uninstalled that. It looks like I'm just getting the 429 message. So, maybe it's still chewing on them.

[brianpetro]

Thanks for sharing that log. It does look OK.

If you don't mind sharing, how many total notes are in your vault?

Also, if any folders have a lot of notes that aren't high-priority for you right now, you can exclude those using the file or folder exclusion settings. That may speed things up.

Lastly, I'm making note of a feature that would bring more visibility into the initial embedding process. Something like "Initial Processing: Currently completed X/Y000 embeddings..."

[mattgrieser]

2904 files in 274 folders. Thanks for the tips. I'll keep an eye on things!

[brianpetro]

My vault has 1,500 files and embeds fully in less than a minute. And another user has a working vault size >20K. So your vault size shouldn't be the issue.

Are there files with a disproportionate amount of content and, more specifically, many headings? This could explain the excessive 429s, which could be causing other things to act unexpectedly and halt the entire process.

Try excluding those files first to see if that allows the embeddings to complete.

Another possible features note:

1. optimizing block embedding batching (currently, piggy-backing file-batching which could create excessive batch sizes, and thus 429s, under certain conditions)
2. a manual override to disable note-block parsing, which embeds partial notes for more granular matching.
3. better queuing during the initial/bulk processing to prioritize a first pass-through of all the notes before embedding with more granularity.

[Eleksar387]

My vault has just 178 files but I also do get 429s and no results after updating to 1.0.36 I have a handfull of images in it as well. Could this be a problem?

[brianpetro]

Hi @Eleksar387

In 1.0.37 the max possible number of simultaneous HTTP requests, the cause of 429s, was reduced from 40 to 20.

Do some of those 178 files have a lot of content, particularly with many headings? If so, then this should help.

Images should not be a problem. I also have many images in my vault.

And thanks for the screenshot!

[Eleksar387]

I am now on 1.0.38. I do not have any overly large files with lots of headings. Rather short and simple. Unfortunately I still see just "Making smart Connections...". Even after hours.

But there is a new error:

[brianpetro]

@Eleksar387 I fixed that error (a function scoping bug/oversight) in 1.0.39.

Also, I added the skip_sections feature/flag in 1.0.40.

Toggled on, skip_sections will limit the max simultaneous API requests to 4, which should altogether remove 429 errors.

While I don't recommend using this feature, as it significantly limits the plugin's capabilities, it may help us debug your issue. For example, if you turn it on and it completes making the connections, we can infer that the problem you're encountering is somewhere within the code that breaks notes into sections.

[Eleksar387]

After updating to latest version. I get new kind of errors:

[brianpetro]

Does that happen even with the skip_sections flag on?

[Eleksar387]

Unfortunately yes 😢

[brianpetro]

When you toggle on the log_render flag, does anything new appear in your console?

It should add something that looks like this:

That could help me figure out what's happening.

[Eleksar387]

Strange enough, no it doesn't make any difference.

[brianpetro]

If you could send your vault folder, minus any sensitive notes, to REDACTED then I will see if I can reproduce the issue.

And thanks for your patience and continued help with this!

[Eleksar387]

I have to thank you! Will do. Probably not today but tomorrow.

[brianpetro]

Sounds good. Make sure to ping me here when you do so it doesn't get lost in my inbox. I'm removing my email from the above response, but you should have it in your email if you've been getting email notifications. And you can let me know if you need it again, just not trying to make it too easy from the scrapers lol.

[Eleksar387]

@billpetro I just played around with a copy of my vault. I removed all of plugins beside (BRAT and smart connections). Also removed a lot of files. It is still showing 429s but now I do get the message "Error getting embeddings for XYZ.md" instead of just "Making Smart connections".

Just sent you a copy of the fault.

[brianpetro]

There's something wrong with your API key.

I got the same error at first. But everything looked fine. So I tried my API key.

When I put my API key in, it worked right away. Full vault embedded in less than 10 seconds.

[Eleksar387]

Holy cow! You are right. How embarrassing. I am sorry for all your hassle! It works.

[quangnd159]

Sorry for digging this up but the same problem is happening to me: retrying request (429) in 1 second... I'm on macOS Ventura, Obsidian 1.1.12. Tried different API keys but still got the same message. And on the sidebar it only says "Making Smart Connections..."

[brianpetro]

@quangnd159 Can you toggle on render logs and render logs files? https://github.com/brianpetro/obsidian-smart-connections/issues/13#issuecomment-1382184147

Then disable and re-enable the plugin.

Are there any logs or errors in the console log? https://github.com/brianpetro/obsidian-smart-connections/issues/13#issuecomment-1379102407

And thanks for letting me know about this!

[quangnd159]

Followed your instruction but the problem persists. This is all there is in the console log 👇

[brianpetro]

How many files are in your vault? And are there any with a disproportionate amount of headings? I've put measures in place to handle these, as we have users with 20k notes and I personally have long notes with many headings. But it would still be worth noting.

With that many 429s, it seems like every call to the OpenAI API is failing. So I'm wondering if not the API key then what else. Maybe you could check to make sure there aren't any preceding or trailing space characters in your API key input, which is something I'm now noting to do automatically in a future update.

You can send me your vault (minus any sensitive files) and I can try to debug it from there. My email is brian@plexm.com (but ping me here if/when you send it).

And thanks again for your help figuring this out. The plugin is supposed to work out of the box for anyone, so solving the edge cases is important to me!

[brianpetro]

I'll also be shipping a "test API key" button shortly. Which should help us rule that out easier.

[brianpetro]

@quangnd159 just pushed an update with the test API key button in the settings

[OntologicallyUnstable]

Hey Brian, same problem here. Vault with 1175 notes; tested yesterday with a fresh re-install without plugins, and just now with the "test API key" function.

[brianpetro]

This from OpenAI seems to imply varying rate limits between accounts.

So I think next step would be to implement exponential back off, as recommended, and see if that helps with moving through these 429s.

[OntologicallyUnstable]

I'll be glad to make another attempt as soon as you pull another update!

[brianpetro]

@OntologicallyUnstable latest update handles errors much better. Also skips previously failed files until retried from the settings

[quangnd159]

@quangnd159 just pushed an update with the test API key button in the settings

Same problem so far. And when I hit Test API Key, it indeeds says the key "is not working as expected".

[brianpetro]

@quangnd159 if you see that, then your API key isn't working and the plugin will not work for you. You will have to contact OpenAI to figure out why.

[OntologicallyUnstable]

@brianpetro Success! Now it worked. Took almost 20 minutes and the log is full of errors, but it worked indeed. Do you know if it's possible to use this tool to automatically update the content of a set of notes, like a series of MOCs or indexes?

[quangnd159]

@brianpetro Thanks a lot. I'll wait and check back at some point.

[brianpetro]

@OntologicallyUnstable happy to hear it!

Are you looking for something like in this request https://github.com/brianpetro/obsidian-smart-connections/issues/17#issue-1533235414 ? If so, add your input to give it some priority.

Currently, you can "drag and drop" the links onto the page. But otherwise the links aren't saved permanently anywhere right now.

[OntologicallyUnstable]

@brianpetro thanks, I'll stay tuned for future updates and provide data in case of new bugs. Keep up the good work, this plugin definitely has a lot of potential!