search

n4ze3m / dialoqbase

Create chatbots with ease
https://dialoqbase.n4ze3m.com/
MIT License
1.6k stars 264 forks source link

Github #281

Closed djav1985 closed 2 months ago

djav1985 commented 2 months ago

I see in change log you added new GitHub loader. What is the exact change? is it more capable of larger repos? if so any rough deal what it can handle?

n4ze3m commented 2 months ago

Yes, the current one can handle a larger repo. I tested it on MinIO Docs with around 700+ files. It may take some time to create the embeddings.

For the embedding model, I tested Nomic via Ollama and text-ada embedding from OpenAI. I do not recommend using all-minilm for larger repo

djav1985 commented 2 months ago

What about the two newer text embedding models the small and large one from open AI?

n4ze3m commented 2 months ago

It will work fine, I mean, in case you want to use an embedding model running locally