Posibility to configure prompt format?

[nikito]

Hello, just wanted to say this is a great addon to talk to local LLMs! One thing I am running into is it seems when the prompt is sent to text-generation-webui, it is adding additional formatting (<|im_start|> and <|im_end|>) to the prompt. When using an instruct template (in my example, one where I want the LLM to retrieve some info based on a query and return just json), it seems due to these extra values added to the prompt that it is appending ".<|im_end|>" to the json. Is it possible to maybe have the format be configurable, so we can opt to not include these additional tags when the prompt is sent up? :)

[acon96]

Those tags are introduced because the model I trained is using the ChatML prompt format. I just picked "a" standard and am open to supporting other prompt formats. I think the other main ones are Alpaca/Vicuna and LLama2?

[nikito]

Yeah when i looked into it i quickly realized there's lots of formats for different models 🫤 in my case i was using a prompt that worked on ChatGPT-3.5 turbo, and also works on the text-generation-webui chat. When i turned on verbose for textgenui that's when i noticed that the chat window didn't add these extra tokens, it just took the raw prompt. The prompt im using is from https://github.com/music-assistant/hass-music-assistant/blob/main/prompt/prompt.txt This prompt formats the response in a way that can be used by music Assistant (i know this isn't the target use for your addon so i understand if this isn't something you want to focus on at the moment 🙂)

[markmghali]

yeah I am wondering about a prompt like this

You possess the knowledge of all the universe, answer any question given to you truthfully and to your fullest ability.
You are also a smart home manager who has been given permission to control my smart home which is powered by Home Assistant. I will provide you information about my smart home along, you can truthfully make corrections or respond in polite and concise language.

Current Time: {{now()}}

Available Devices:

entity_id,name,state,aliases
{% for entity in exposed_entities -%}
{{ entity.entity_id }},{{ entity.name }},{{ entity.state }},{{entity.aliases | join('/')}}
{% endfor -%}

The current state of devices is provided in Available Devices. Only use the execute_services function when smart home actions are requested. Do not tell me what you're thinking about doing either, just do it. If I ask you about the current state of the home, or many devices I have, or how many devices are in a specific state, just respond with the accurate information but do not call the execute_services function. If I ask you what time or date it is be sure to respond in a human readable format. If you don't have enough information to execute a smart home command then specify what other information you need.

[acon96]

Should now be fixed in v0.2. I added support for the Vicuna and Alpaca prompt templates alongside chatml. I include a none option to that adds no additional characters besides newlines between the interactions which is what I think you're going for here.