home-assistant / core

:house_with_garden: Open source home automation that puts local control and privacy first.
https://www.home-assistant.io
Apache License 2.0
71.65k stars 29.94k forks source link

Failure to manage home with Ollama when aliases for entity exist #126391

Open Rudd-O opened 2 hours ago

Rudd-O commented 2 hours ago

The problem

Ollama correctly analyzes a question like "is the office ceiling light on?" (responding it's off).

But when asked to turn it on, here is the full debug log (the relevant snippet) from Ollama:

You are a helpful assistant with tool calling capabilities. When you receive a tool call response, use the output to format an answer to the orginal use question.<|eot_id|><|start_header_id|>user<|end_header_id|>

is the office ceiling light on?<|eot_id|><|start_header_id|>assistant<|end_header_id|>{"name": "HassTurnOff", "parameters": {"domain":"light","name":"Office ceiling light, luz de la oficina"}}{"name": "HassTurnOn", "parameters": {"domain":"light","name":"Office ceiling light, luz de la oficina"}}<|start_header_id|>ipython<|end_header_id|>

{"error": "MatchFailedError", "error_text": "<MatchFailedError result=MatchTargetsResult(is_match=False, no_match_reason=<MatchFailedReason.NAME: 1>, states=[], no_match_name=None, areas=[], floors=[]), constraints=MatchTargetsConstraints(name='Office ceiling light, luz de la oficina', area_name=None, floor_name=None, domains={'light'}, device_classes=None, features=None, states=None, assistant='conversation', allow_duplicate_names=False), preferences=MatchTargetsPreferences(area_id=None, floor_id=None)>"}

{"error": "MatchFailedError", "error_text": "<MatchFailedError result=MatchTargetsResult(is_match=False, no_match_reason=<MatchFailedReason.NAME: 1>, states=[], no_match_name=None, areas=[], floors=[]), constraints=MatchTargetsConstraints(name='Office ceiling light, luz de la oficina', area_name=None, floor_name=None, domains={'light'}, device_classes=None, features=None, states=None, assistant='conversation', allow_duplicate_names=False), preferences=MatchTargetsPreferences(area_id=None, floor_id=None)>"}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

No, the office ceiling light is not on.<|eot_id|><|start_header_id|>user<|end_header_id|>

turn it on<|eot_id|><|start_header_id|>assistant<|end_header_id|>{"name": "HassTurnOn", "parameters": {"domain":["light"],"name":"Office ceiling light, luz de la oficina"}}<|start_header_id|>ipython<|end_header_id|>

{"error": "MatchFailedError", "error_text": "<MatchFailedError result=MatchTargetsResult(is_match=False, no_match_reason=<MatchFailedReason.NAME: 1>, states=[], no_match_name=None, areas=[], floors=[]), constraints=MatchTargetsConstraints(name='Office ceiling light, luz de la oficina', area_name=None, floor_name=None, domains={'light'}, device_classes=None, features=None, states=None, assistant='conversation', allow_duplicate_names=False), preferences=MatchTargetsPreferences(area_id=None, floor_id=None)>"}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Pay close attention to the last two lines. It's attempting to set the name "Office ceiling light, luz de la oficina" which is very clearly not a light at home, hence the MatchFailedError. That makes sense, since it was given that "name" by the integration itself in the system prompt:

- names: Office ceiling light, luz de la oficina
  domain: light
  state: 'off'
  areas: Office

Given that Ollama (LLAMA 3.1) is failing to understand the nuance of names vs. name, then the YAML prompt given to Ollama has to be reengineered to separate the aliases from the name, rather than unifying them both in a single field. Consider this as an alternative:

- name: Office ceiling light
  also known as: Luz de la oficina

Alternatively, we the users should be given the ability to customize the template again (this feature was taken out a few releases ago).

What version of Home Assistant Core has the issue?

core-2024.9.0

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant Core

Integration causing the issue

ollama

Link to integration documentation on our website

No response

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

home-assistant[bot] commented 2 hours ago

Hey there @synesthesiam, mind taking a look at this issue as it has been labeled with an integration (ollama) you are listed as a code owner for? Thanks!

Code owner commands Code owners of `ollama` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign ollama` Removes the current integration label and assignees on the issue, add the integration domain after the command. - `@home-assistant add-label needs-more-information` Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue. - `@home-assistant remove-label needs-more-information` Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

(message by CodeOwnersMention)


ollama documentation ollama source (message by IssueLinks)