Open Juxsta opened 1 month ago
For an MVP a possible solution could be to allow a custom prompt parameter where you can pass in a template and variables similar to the way the current aiuser prompt setting works
From what I understand, the Jinja template is to mostly translate to a format more suitable for instruct models?
Does existing OpenAI proxy servers not do that type of translation for instruct models already?
I don't believe this is a supported feature since you would need to be able to pull variables from the prompt to support it. This would be possible if the prompt was constructed in such a way the templating could be done through the proxy, but at that point, the templating may as well have been done on the client.
Just for clarity the feature request I'm proposing is being able to make a setting such as
template: -|
<s> [INST] You are {bot_name}. You are responding in a discord chat room in the {discord_server} in the {discord_channel} channel.
Your personality traits are:
{Persona}
[/INST]
</s>
Message History:
{Message History}
[INST] Respond as {bot_name} [/INST]
{bot_name}:
Ah, I understand now.
Currently, the cog is not greatly compatible with instruct models (mostly because I found chat models better to chat with and easier to prompt with). I'll see if I can add this feature whenever I get to improving compatibility with instruct models.
Would it be possible to supply an option for using an instruction template using Jinja? This would significantly improve the performance of instruct models.