search

acon96 / home-llm

A Home Assistant integration & Model to control your smart home using a Local LLM
483 stars 56 forks source link

Sorry, there was a problem talking to the backend: 'text' #137

Closed StefanDanielSchwarz closed 2 months ago

StefanDanielSchwarz commented 2 months ago

Describe the bug
I'm trying to use the chat completion HTTP endpoint of oobabooga's text-generation-webui instead of the text completion one. This gives this error message as the assistant's output in Home Assistant:

Sorry, there was a problem talking to the backend: 'text'

By the way, text completion works, but since there's no prompt template for Command R+ or Phi-3 and apparently also no proper stopping sequence defined, I need to use chat completion.

Expected behavior
Inference should work with chat completion like it does with text completion.

Logs
I see in ooba's textgen console that the correctly formatted prompt arrives. And I see it generating a response:

Output generated in 12.47 seconds (8.58 tokens/s, 107 tokens, context 3485, seed 1187181536)

It seems to just not arrive back in the HA assistant.

I'm an AI engineer who got into HA because of AI integration, so not an expert with the HA parts yet, let me know if you need further information or how I could troubleshoot this myself. I'd also be happy to provide the prompts for the aforementioned models in a PR.

Update: This was with the Generic OpenAl Compatible API. After switching to the text-generation-webui API, it works! Prompt and stopping works now.

It's up to you if we consider that a solution or if it's worth investigating why it doesn't work with the OpenAl API. But I'm happy to report it's working now and I can concentrate on the prompt engineering to make this work with HA and Command R+.

acon96 commented 2 months ago

Yes that is the reason that text-generation-webui is split out in a different backend. It returns objects that have a different object type than what the OpenAI spec says.