search

acon96 / home-llm

A Home Assistant integration & Model to control your smart home using a Local LLM
491 stars 56 forks source link

"I'm sorry, but I don't have the ability to control light switches." #66

Closed brando-online closed 4 months ago

brando-online commented 4 months ago

I followed the walk through on this video: https://www.youtube.com/watch?v=fNuHVok_hWY

Unfortunately, I could not use the v2 of the model located at the following URL; there was an initial loading error on the Oobabooga web UI, no matter what I tried or which computer I attempted to load it with: https://huggingface.co/acon96/Home-3B-v2-GGUF

So I had to use the v1 model - loaded without a hitch: https://huggingface.co/acon96/Home-3B-v1-GGUF

So I get it all configured in HA and then ask it to turn on living room lights and get the following response: I'm sorry, but I don't have the ability to control light switches.

I am running Oobabooga on a secondary computer with a GPU to keep responses quick, and then accessing it via HA through Oobabooga's API. I do have over 200 entities exposed to the Voice Assistant in HA Settings, so I'm not entirely sure why it wouldn't have the ability to control lights.

I can confirm that if I switch over to the built in Home Assistant voice assistant, the command works and all of my living room lights do turn on; just not with the Home LLM integration.

Home Assistant version 2024.1.6 HACS 1.34.0 Debian v12 Bullseye

Any ideas on what could be causing this or pointers to other possible issues?

acon96 commented 4 months ago

I do have over 200 entities exposed to the Voice Assistant in HA Settings, so I'm not entirely sure why it wouldn't have the ability to control lights.

This is almost certainly causing the system prompt instructions to be excluded form the model's context due the massive size of the list of devices you have exposed. For now you should stick to using under 50 devices (or even fewer if you can) or you can try to use the solution from this duplicate issue until there is a better solution: https://github.com/acon96/home-llm/issues/55

brando-online commented 4 months ago

I removed ALL exposed devices and only 1 single device to try and control it, as well as updated the GBNF grammar. Still failed the same way. I'll wait until this is perfected a bit more since you closed the issue out without confirming. Thank you anyway for your time.

acon96 commented 4 months ago

I closed the issue because from what you have said: it is a setup issue, not a bug with the component. The model is being prompted incorrectly and causing it to hallucinate. If you need help configuring it correctly please open a discussion instead of an issue.