Latest HA 2024.8.0 with latest Ollama and pretty much every model is unusable

[Rudd-O]

The problem

Whenever I ask the model to give me the state of any of the entities listed in the preamble sent to Ollama, the response from the LLM is basically trash.

what is the state of the guest bathroom lights?

{"name": "HassLightGet", "parameters": {"device_class": "["}"} Please note that I've assumed a function named "HassLightGet" exists to get the state of light devices. If it's not present in your original list, please let me know and I can try again or we can create a new function call.

This is actually worse than the previous version. In the previous version, the model would at least recognize some of the entities in the listing sent within the preamble. Now the model just hallucinates entities that don't exist, and refuses to identify / describe / control them.

I'm using llama3.1:8b and also mistral-nemo:latest. Same result with both models. I've verified with tcpdump that the calls are correct and the responses are flowing back to HA.

In every way this is a massive regression.

I also can't alter the entities part of the prompt by configuration either. The default prompt is basically a paragraph, and all the templated entity stuff that used to be in the configurable prompt is gone -- it is replaced by something which HA is adding on its own without any configurable (templated) content. (Plus, of course, the tools JSON.) I can see it all in the logs and in tcpdump output.

What version of Home Assistant Core has the issue?

core-2024.8.0

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant Core

Integration causing the issue

ollama

Link to integration documentation on our website

No response

Diagnostics information

debuglog.txt

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

[Rudd-O]

LOL.

[Rudd-O]

I have done some more debugging. It appears that when I send the exact same text (or nearly the same text, I explain to the model that the entities list is formatted in YAML), directly from Open-WebUI, I can get valid informational responses (although it doesn't seem to recognize some entities, no doubt due to default context token size, for which I opened #119946 a while ago).

So, it's almost certain that something in the Ollama side is totally ignoring the role: system prompt. Not sure how this will need to be fixed, but it does seem like that's broken right now and, as a result anything that uses a system prompt prior to the role: user prompt simply won't work properly.

I literally updated Ollama today. Sigh.

[Rudd-O]

Possibly related? https://github.com/ollama/ollama/issues/6176

[Rudd-O]

Yeah so I have confirmed that the issue is in latest ollama.

I'm closing this.

[Rudd-O]

I figured it out! I figured it out. When the system prompt is above a certain length (in my case, about half the text of what HA generates), it is totally ignored. Halve the text or double the context token size and BAM, suddenly it works!

Once again, I must ask that the context token size be configurable https://github.com/home-assistant/core/issues/119946 or that it is dynamically increased proportional to the amount of system prompt text. The fix is here: https://github.com/home-assistant/core/pull/121803 .

[allenporter]

Something to consider here is that you may want to try playing with a model with and without assist. From https://llama.meta.com/docs/model-cards-and-prompt-formats/llama3_1 Note: We recommend using Llama 70B-instruct or Llama 405B-instruct for applications that combine conversation and tool calling. Llama 8B-Instruct can not reliably maintain a conversation alongside tool calling definitions. It can be used for zero-shot tool calling, but tool instructions should be removed for regular conversations between the model and the user.

This is a contributing factor to the above behavior your are seeing, so just sharing for context for now.

[weberruss]

I myself recognized that the language setting is also totally ignored. I have configured german but assist with llama is ignoring that and is always speaking in english with me. Also it is totally unable to control things. It seems that he does not know that if i am taking about lights, that he needs to control entities of the domain light. In addition he cant handle unavailable devices at all. As soon there is a single unailable device he is stopping his work at all. Totally useless right now. All of that works fine with chat-gpt-4.0-mini

[home-assistant[bot]]

Hey there @synesthesiam, mind taking a look at this issue as it has been labeled with an integration (ollama) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `ollama` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign ollama` Removes the current integration label and assignees on the issue, add the integration domain after the command. - `@home-assistant add-label needs-more-information` Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue. - `@home-assistant remove-label needs-more-information` Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

_{^{(message by CodeOwnersMention)}}

---

ollama documentation ollama source _{^{(message by IssueLinks)}}