search

QuivrHQ / quivr

Open-source RAG Framework for building GenAI Second Brains 🧠 Build productivity assistant (RAG) ⚡️🤖 Chat with your docs (PDF, CSV, ...) & apps using Langchain, GPT 3.5 / 4 turbo, Private, Anthropic, VertexAI, Ollama, LLMs, Groq that you can share with users ! Efficient retrieval augmented generation framework
https://quivr.com
Other
36.17k stars 3.51k forks source link

[Feature]: quivr with local llm #1611

Closed robertsmaoui closed 9 months ago

robertsmaoui commented 10 months ago

The Feature

In my opinion, it will be more interesting if quivr can be used with offline models without the need for OpenAI api key,

Is this possible with the new versions?

Motivation, pitch

this feature saves money and allows chat confidentiality !

Twitter / LinkedIn details

No response

dosubot[bot] commented 10 months ago

🤖

Hello @robertsmaoui,

Nice to meet you! I'm a bot here to assist with your inquiries, help with bug fixes, and guide you on contributing to our repository. While you're waiting for a human maintainer, I'm here to provide any help I can. Let's make the most of our time together!

It seems like you've got this under control, if you want help or have specific questions, let me know what I can do for you!

StanGirard commented 10 months ago

https://www.quivr.app/blog/is-mistral-a-good-replacement-for-openai ;)

robertsmaoui commented 10 months ago

  1. In the demo https://www.quivr.app/, it does not work when you choose huggingface/mistralai/Mistral-7B-Instruct-v0.1 (no response in the chat)

  2. In the source code, I did not find any file that mentions the use case of huggingface or environment variables for the endpoint

  3. As long as we depend on the huggingface API it is not considered offline

Thank you very much StanGirard for Quivr

cccc11231 commented 10 months ago

Hi @robertsmaoui

Did you try this: https://brain.quivr.app/docs/Developers/selfHosted/run_fully_local

I am also looking for completely offline version. Do you know does the above instruction work?

I try to deploy quiver locally a few month ago, at that time, everything works except embedding part which still rely on openAI. I don't know current status. Do you know if we can deploy it completely offline or not?

Thanks.

github-actions[bot] commented 9 months ago

Thanks for your contributions, we'll be closing this issue as it has gone stale. Feel free to reopen if you'd like to continue the discussion.

StanGirard commented 9 months ago

@cccc11231 This is now working :)

Japhys commented 3 days ago

@StanGirard

@cccc11231 This is now working :)

I know this was closed but I found this when looking for some answers. the links to the docs do not work and I can't find any documentation on how to use hugging face local models. I am considering using quivr but I am not sure it it fits my need. I want to chose a local multilingual transformer model from hugging face for my embeddings but want (for now) use openai api for the generation part. Is this at all possible in an easy to configure way with quivr?