Ollama model success?

[ericvrp]

I tried various models (Gemma, codellama:7b-instruct, Mistral) with Ollama, using ngrok to be able to access Ollama on another computer then my Raspberry Pi running Home Assistant.

It kind of works but the models don't seem to understand what they have to do. Anyone else give Ollama and this HA integration a try?

[eponce92]

I tried with LM studio and got the same results. Cool for funny conversations if you use uncensored models and a creative system prompt but not good enough for anything usefull. Check out Home LM repo, they have a custom 3B LLM model for local home control that kindda works: https://github.com/acon96/home-llm#home-assistant-component https://huggingface.co/acon96/Home-3B-v3-GGUF

[schoren]

I'm facing the same issue. The HomeLM thing is not working very well either, and I'd like to not only be able to run HASS commands, but also to do general AI questions. I'm using LM Studio on Mac. The speed is awesome, but I don't know how to correctly enable the "functions" featuer

[aflores29]

I tried various models (Gemma, codellama:7b-instruct, Mistral) with Ollama, using ngrok to be able to access Ollama on another computer then my Raspberry Pi running Home Assistant.

It kind of works but the models don't seem to understand what they have to do. Anyone else give Ollama and this HA integration a try?

I am also having this issue in HA. I can get basic chat to work but no device control with ollama.

[neowisard]

Ollama doesn't have any Function calling and tools support as https://github.com/ollama/ollama/blob/main/docs/openai.md

103

[FutureProofHomes]

Agree with @neowisard - Ollama does not yet support OpenAI Function Calling. This is confirmed in ollama documentation (which @neowisard shared above) and in their blog post .

Seems there are a few options:

1. Wait for Ollama to add support for this feature (looks like this is coming in the future).
2. Roll your own solution with LangChain. Has anybody has luck with this strategy? I think this is what LocalAI is doing under the hood, but their implementation hasn't performed well in my testing w/ the Extended_OpenAI_Conversation, and I've tried many models.

Does anybody else see any other immediate paths forward?

Would love to hear your thoughts @jekalmin. Thanks all!

[Terries13]

I'm having connection issues trying to add a local model to extended_openai_conversation. Based on the messages above, it seems like others have been able to actually connect a local model to extended_openai_conversation, although bad results.

I've tried ollama, liteLLM, LocalAI running multiple different models(mistral 7b and llama2-uncensored for ollama) and still getting Failed to connect errors. When I test out functionality of the wrappers with curl, I get a response in the command line and everything looks fine.

I'm running a Pi 5 with updated HA V12.0, Updated Ollama V0.1.27 in WSL2. Windows firewall logs and my router firewall logs don't show any blocked connections. I'm at a loss and spent too many hours at this point. Any advice based on this information?

[Anto79-ops]

@Terries13 try this integration with this model using LocalAI.

https://github.com/jekalmin/extended_openai_conversation/issues/17#issuecomment-1973952771

and the model im reffering to is the q6 version of https://huggingface.co/TheBloke/laser-dolphin-mixtral-2x7b-dpo-GGUF

[jekalmin]

@FutureProofHomes

Would love to hear your thoughts @jekalmin. Thanks all!

I'm also stuck in where everyone else is at. If no model works, fine-tuned model based on HA might be the next step to try (I'm not certain though).

I hope someone who is an expert at this area gives a try.

@Anto79-ops Thanks for continuing your interest in LocalAI. I would try the model you suggested. (The model you previously suggested didn't work well for me. It might be the issue in my environment.)

[jmtatsch]

there is also now a dolphin mistral with a bit of function calling support https://huggingface.co/dagbs/fc-dolphin-2.6-mistral-7b-dpo-laser-GGUF

[Anto79-ops]

thanks! going to try this new model. The original laser dolphins woks pretty well so i'm very curious to see how this one works. cheers

[Anto79-ops]

@jmtatsch I just tried this model q8 with LocalAI. A couple of things I wanted to mention:

1. It worked well in controlling my devices and turning lights on or off. But it was not the greatest at chatting about my sensors, like of a level is too high or low...I suspect it's not as good as a chat model because of point 2

2. The model is noticeably smaller in size compared to the original laser (, so these models might Infact be a 3.5 b based on the model size).

This could explain why it would sometimes repeat the question instead of talking about. You may not experience this because our setup is different. For example, if I have more entities exposed, it might make the model glitch

But it does perform the service functions well.

[jmtatsch]

@Anto79-ops laser-dolphin-mixtral-2x7b-dpo-GGUF is a mixture of experts model containing two 7b models thus its about twice the size of fc-dolphin-2.6-mistral-7b-dpo-laser-GGUF.

[Anto79-ops]

@jmtatsch you're right, I just noticed it's a 7b nevermind my last comment then. It's acting as smart as a 7b is, and does the functions well.

[Anto79-ops]

There is some good news with Home Assistant. It now has Ollama support, but as others have reported, I can't get this integration to talk to Ollama.

[sorgfresser]

In order to get this integration to work with Ollama I would advice using lite-llm and changing the DEFAULT_CONF_BASE_URL to the lite-llm proxy.

[ieaves]

I was able to get this integration working with Ollama using the following procedure:

On the setup page after adding a new extended_openai_conversation integration

1. For the url to your local Ollama instance use the /v1 path e.g. http://0.0.0.0:11434/v1
2. Add anything for the API Key
3. Check "ignore authentication" when creating the connection

Now you'll need to edit the configuration setting for the resulting object.

Look for the "model" field on the configuration page. It will default to something like gpt-3.5. Edit that text field to instead reference whichever model you're using in Ollama (e.g. mistral:latest, llama:8b, etc...)

[eponce92]

will this and llama 3.1 work now using tools ? https://ollama.com/blog/tool-support