search

taketwo / llm-ollama

LLM plugin providing access to local Ollama models using HTTP API
Apache License 2.0
160 stars 10 forks source link

Support for remote ollama server #13

Closed dpatschke closed 4 months ago

dpatschke commented 4 months ago

It looks like there is currently only support for utilizing ollama servers hosted on the default server/port (localhost:11434).

The python API supports from ollama import Client where Client can then be utilized like:

client = Client(host='http://ollama:11434')
client.chat( ...)

vs. the current method of just using ollama.chat.

I'm trying to build a docker-compose application that has an application leveraging llm in one container and ollama server running in another and am facing some headwinds implementing this without the Client functionality readily exposed in llm-ollama.

It seems like this would be very doable. The main question would be where/how to pass in an optional host parameter for Client.

taketwo commented 4 months ago

Perhaps some of the other llm plugins faced a similar problem and has a solution that we can copy?

dpatschke commented 4 months ago

I'll look into it.

My first thought would be to pass a host parameter in via options and then extract and remove it before the options are then passed into the chat method, but this doesn't seem overly elegant.

dpatschke commented 4 months ago

Actually, I found a solution.

The following line of code suggest that adding a OLLAMA_HOST environment variable should solve the problem. https://github.com/ollama/ollama-python/blob/8b694bb0f4578c07ab50da06d80bc7276a470e7c/ollama/_client.py#L56

I did, in fact, test this and it works. Closing the issue.

dpatschke commented 4 months ago

Actually, I found a solution.

The following line of code suggest that adding a OLLAMA_HOST environment variable should solve the problem. https://github.com/ollama/ollama-python/blob/8b694bb0f4578c07ab50da06d80bc7276a470e7c/ollama/_client.py#L56

I did, in fact, test this and it works. Closing the issue.

taketwo commented 4 months ago

Thanks for researching. I've added a note about this variable to the README.

regularfry commented 1 month ago

I'm not sure what changed and when, but this doesn't work today. Or at least, doesn't work for llm ollama list-models. The problem is that the ollama library instantiates a bare Client() to use by default here which ignores the environment variable, and that's what's used when you call ollama.list here. The result is that the available models never get registered, so you can't select them for prompting.

taketwo commented 1 month ago

I've followed your first link and I think this bare Client() actually gets host set from environment variable, see here: https://github.com/ollama/ollama-python/blob/ebe332b29d5c65aeccfadd4151bf6059ded7049b/ollama/_client.py#L57

regularfry commented 1 month ago

Ah you're right, of course. I'm being misled by IPv6 horribleness.