search

pfrankov / obsidian-local-gpt

Local Ollama and OpenAI-like GPT's assistance for maximum privacy and offline access
MIT License
340 stars 24 forks source link

REQ: Local Rag Support #55

Open iamhenry opened 4 days ago

iamhenry commented 4 days ago

Local GPT is great and i love what you've built

I use it for journal entries and was wondering if you can integrate a simple RAG solution?

When i write my journal entries i look for :

I have a custom system prompt that analysis my current note, but would love to be able to use RAG to reference all my notes for a better assessment

I would use an embeddeding from Ollama

image
pfrankov commented 3 days ago

Thanks for the suggestion! The plugin already has RAG but in a different, more useful way:

All the links (and their links) that you include in a selection would be processed as a context for RAG.

Why is that

Unfortunately there is no point to include a lot of documents as a context, especially for OpenSource LLMs due to Needle In a Haystack problem — most local LLMs can not truly operate more than 8192 tokens, even if they claim so. That means the overall result will be better in shorter context and worse in larger. This is why we can not use RAG on entire vault.

iamhenry commented 3 days ago

I was aware but thanks for clarifying. I wasn't sure how to use it

So because of the limitation of the local models it's not currently possible?

I'm curious to hear your thoughts on how the plugin Smart Composer is able to accomplish it?

I'm not able to get the embed model working with that plugin so I prefer to use Local GPT since it automatically detects the models for me

https://github.com/glowingjade/obsidian-smart-composer

pfrankov commented 1 day ago

I'm curious to hear your thoughts on how the plugin Smart Composer is able to accomplish it?

It doesn't. If you have embedded entire vault and take only 10 chunks from it it's completely a blind luck to get useful results.

But I must say that there is a difference between Smart Composer and Local GPT: Smart Composer has fuzzy search (like regular search) and Local GPT doesn't have yet. This could be a little boost to the quality in some cases but not for all.

As I've said Local GPT has the best way to work with local documents for local models.