Whole Vault is Scanned but AI seems unable to recall anything to do with it...

[yannlk]

The conversations I have with the AI are "useless" I have over 2'000 files of context and no failed files so I don't understand why it doesn't work.

Edit: An example of a conversation that should definitely yield something based on the context:

[yannlk]

Now GPT is saying it has only 10 notes of context, did I set it up wrong or something? It definitely worked for more because it charged 2$ on my openai account.

I also have no excluded files or failed files.

[brianpetro]

@yannlk thanks for helping me with this!

It's tough to say what might be happening without further understanding what your notes look like.

What is your folder/file structure like? Do you use headings in your notes?

The current version sort of depends on those two, so a zettelkasten setup might not perform as well. However, this should be improved in time.

Personally, I've been impressed by the utility of this feature. Of course, the current design is biased towards my personal style. But I definitely think that the code just has to catch up with all the different ways people format their notes.

Thanks for the screenshots and any insights you can provide that will help me figure out how to make this work better for everyone!

[brianpetro]

Now GPT is saying it has only 10 notes of context, did I set it up wrong or something? It definitely worked for more because it charged 2$ on my openai account.

I also have no excluded files or failed files.

@yannlk the ten notes of context is just how many were included as context in the response, all your notes are still indexed as embeddings. Sorry for the confusion.

[abdcef]

Hi Brian, thanks for making this plugin. Something that I have wanted to try since the AI hype started :) I have a similar difficulty as the OP and would like to know if you could suggest some dos and don'ts for note organization for this plugin to work well. My daily notes are in folders with several atomic notes per day. And AI has difficulties using them as context even though each atomic note for that day includes %%[[YYYY-MM-DD]]. It runs well on some day notes and completely ignores others. Knowing what can help make it respond well will be great.

Look forward to seeing what's next :)

[brianpetro]

@abdcef thanks for the screenshot. That helps a lot.

So, right now, the plugin is optimized for named file/folder systems. I realize this excludes popular date-based and zettelkasten systems, which are also part of my PKM system, so I have been thinking about solutions.

In addition to file and folder names, the plugin puts a lot of weight on named headings within your notes. This may be why some of your notes perform better than others.

So notes with descriptive file, folder, and heading names will perform better with the plugin at this time.

But, thanks to AI, you won't necessarily have to go and reorganize your notes. I'm considering fine-tuning a model based on my context-subject framework that would fill in the contextual gaps in date-based/zettelkasten notes.

Additionally, if tags or links are a big part of the system, we may be able to utilize them in a similar way.

I hope that helps bring some clarity to the situation for you.

Thanks for your contributing your feedback. And don't hesitate to let me know if you still have questions. Both help me better improve the plugin for everyone.

Brian🌴

[abdcef]

Brian, thank you so much for your clear and helpful reply! I'm really excited about your roadmap and I'm so glad that you have released this plugin for the Obsidian community. It's really appreciated! Thanks again for all your hard work!

I was wondering if there was a way to know when the embeddings have been updated. For instance, if I rearrange some of my notes, will the plugin update the embedings in real-time?

Also, I have three json files in my vault. Does the plugin create multiple embedding files?

[brianpetro]

😊

Embeddings are updated pretty quickly, typically when you open a new note which triggers the "Find connections" command. But some times updates are prevented to avoid increasing OpenAI costs. For example, if the file size isn't changed much (<10%) then update won't be triggered. But changed the file or folder name will trigger the update at the next "Find connections" trigger.

embeddings-2.json is the primary file with your Embeddings.

The following three files can be safely deleted. embeddings-TIMESTAMP.json (second file in list) is generated when you "force refresh" embeddings as a sort of backup. I think the other two are self explanatory.

[sam1am]

I wrote an About Me note in my vault with the basic facts it needs to know about me and my major areas of life - my job, relationships, etc.

I realize my note taking doesn’t have a lot of explicit info making it obvious what is going on.

[abdcef]

I realize my note taking doesn’t have a lot of explicit info making it obvious what is going on.

That's a creative way to let your vault know you more ;) I hope this can make the chat more personal as the plugin improves on its mission to attain sigularity with the note maker.

[yannlk]

Thank you for the explanation @brianpetro. I want to thank you for your work on this, and in testing the chat of the extension, I've realised how useful the "Smart connections" aspect of the plugin is.