How do I run the Ollama [Bug]:

FaizanMunsaf commented 3 days ago

What happened?

I am trying to run Ollama there's no specific command I found that run Ollama private model. There's any further guidance available with current repo?

Relevant log output

Just looking forward to Ollama show into my bot, I will generate my answer using Ollama llama3-8b model!

Twitter / LinkedIn details

No response

linear[bot] commented 3 days ago

TECH-61 How do I run the Ollama [Bug]:

dosubot[bot] commented 3 days ago

Hey @FaizanMunsaf! I'm here to help you with the Ollama issue you're facing. I'm a bot designed to assist with bugs, questions, and contributions. Let's work on resolving this together.

To run the Ollama private model using the current repository, follow these steps:

Install Ollama: Install Ollama from their website.
Run Ollama in the background: Execute the following command to run Ollama:
```
ollama run llama2
```

Run Migrations: Move the migration file and reset the database:

mv supabase/migrations/local_20240107152745_ollama.sql supabase/migrations/20240107152745_ollama.sql
supabase db reset

Add Ollama Model to Quivr: Update the user_settings table in Supabase to include the Ollama models:
```
[
 "ollama/llama2",
 "ollama/mistral"
]
```
If you want to add it to your user only, you can add it to the models column in the user_settings table. If you want to set it as default for all new users, drop the entire table with the following command:
```
DELETE TABLE user_settings;
```
Configure .env File: Update your .env file to activate the Ollama API by uncommenting and setting the OLLAMA_API_BASE_URL:
```
# LOCAL
OLLAMA_API_BASE_URL=http://host.docker.internal:11434
```

Integrate with Quivr: Ensure that the QuivrRAG class in quivr_rag.py is properly configured to use the Ollama model. The ChatLiteLLM class should be instantiated with the api_base parameter set to the OLLAMA_API_BASE_URL if the model starts with "ollama". This is handled in the _create_llm method of the QuivrRAG class:

def _create_llm(
   self,
   callbacks,
   model,
   streaming=False,
   temperature=0,
) -> BaseLLM:
   if streaming and callbacks is None:
       raise ValueError(
           "Callbacks must be provided when using streaming language models"
       )

   api_base = None
   if self.brain_settings.ollama_api_base_url and model.startswith("ollama"):
       api_base = self.brain_settings.ollama_api_base_url

   return ChatLiteLLM(
       temperature=temperature,
       max_tokens=self.max_tokens,
       model=model,
       streaming=streaming,
       verbose=False,
       callbacks=callbacks,
       api_base=api_base,
   )

By following these steps, you will have the Ollama model running and integrated with Quivr ^[1]^[2]^[3].

To continue talking to Dosu, mention @dosu.

roaidev commented 16 hours ago

real problem is getting it to work with llama3. how do we do that?

QuivrHQ / quivr