search

Cinnamon / kotaemon

An open-source RAG-based tool for chatting with your documents.
https://cinnamon.github.io/kotaemon/
Apache License 2.0
17.42k stars 1.34k forks source link

[REQUEST] Make Ollama URL customizable #505

Open ferpizza opened 2 hours ago

ferpizza commented 2 hours ago

Reference Issues

No response

Summary

Currently, Ollama URL is hardcoded to http://localhost:11434/api

This becomes an issue when deploying Kotaemon in more advanced scenarios, like running different containers for Kotaemon and Ollama.

Basic Example

Kotaemon docker image doesn't include Ollama, so one approach to tackle this is to run a 2nd docker image with Ollama in it.

The challenge is that on this scenario, Ollama is no longer available on http://localhost:11434/api, but on some other URL like http://ollama:11434/api.

As Ollama URL is not a setting we can manage through EnvVars, the above deployment is not possible without having to tweak the code.

Drawbacks

We are unable to run Kotaemon and Ollama on different VMs/Containers, and this is a common pattern when working with contaiers (eg. kubernetes, docker-compose)

Additional information

No response

taprosoft commented 2 hours ago

@ferpizza User can always edit any parameters to the models, including base_url like this image Can you try this method?

ferpizza commented 1 hour ago

In my case, the issue with Ollama was happening at the initial Setup screen, where we have no chance to edit these settings ... and it was failing to download models and move forward (and I didn't wanted to go into advance mode to get to the screen you are sharing).

I workaround it by adding a sed command to the official docker entrypoint, and was able to download models and get a working connection with Ollama running on an independent container.

But I think that having this on an EnvVar is better and cleaner. (also, my workaround is not bulletproof)

    entrypoint:
      - /bin/sh
      - -c
      - |
        sed -i 's/localhost:11434/ollama:11434/' libs/ktem/ktem/pages/setup.py &&
        python app.py
ferpizza commented 1 hour ago

Also, on the screen you share, I still see localhost:11434 so I guess it's not showing the actual config used on the instance, but some default setting that are sourced from somewhere else? ... maybe from flowsettings.py (that I havent changed) ?

taprosoft commented 1 hour ago

@ferpizza yes we did not consider alternate Ollama endpoint in first setup screen. User are expected to use Skip setup and Set the model in the UI manually later.

An env var specifically for Ollama might probably help, we will consider this feature.