Template format for local models

[synw]

A nice feature to have would be to be able to select a prompt format. The prompt templates are different for each model family. I see that you use the Alpaca format ( ### Instruction: ... {prompt}\n\n### Response:). For example it could be nice to be able to change the prompt template to for example the Llama 2 format:

<s>[INST] <<SYS>>
You are a helpful, respectful and honest assistant. Always answer as helpfully as possible

If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.
<</SYS>>

{prompt} [/INST]

or Orca 2:

### System:
You are an AI assistant that follows instruction extremely well. Help as much as you can.

### User:
{prompt}

### Response:

[Edit] Example use case: I would like to use https://huggingface.co/TheBloke/LosslessMegaCoder-Llama2-7B-Mini-GGML that looks good on coding tasks, but it's template format is ChatMl:

<|im_start|>system
{system_message}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant

[nvms]

I totally agree. This is another one of the things handled by the lib I use for request/response formatting. Specifically, this happens here:

https://github.com/transitive-bullshit/chatgpt-api/blob/main/src/chatgpt-api.ts#L395

Maybe it's time to roll my own solution for this, so that a prompt formatting can be explicitly defined. I'd like to implement this, but need some time to think about how best to do so.

[synw]

I studied this library recently when trying to implement OpenAi support in my server project. It is easy to replace: example code:

• With the library
• Without the library: using just the eventsource-parser library (used internaly in the chatgpt api lib)

[nvms]

Nice, seems easy enough. I should have time to get this done this evening.

How would you expect to define a template? I think I'd probably surface this as an item in the wingman settings panel as e.g. wingman.openai.template, as well as allow it to be overridden by any given command. Does that seem okay to you?

[synw]

yes it looks good. I would define a template in a very simple maner. An Alpaca one:

const template = `{system}

### Instruction: fix this invalid json:

'''json
  {prompt}
'''

### Response: (answer in valid json)
`

const system = "You are a javascript code specialist. Below is an instruction that describes a task. Write a response that appropriately completes the request";

const _prompt = '{"a": 1,}';

const finalPrompt = template.replace("{system}", system).replace("{prompt}", _prompt)

where {system} is actually covered by systemMessageTemplate in Wingman

[synw]

Note about handling the sse request: the eventsource-parser lib is not really necessary: we can handle this manually using a regex, this is what I did in my frontend: example

[synw]

About templates I just started a little library to manage them more efficiently an use prebuilt template formats: https://github.com/synw/modprompt

I got tired to copy and change text to different template formats depending on the models, so I wrote this to ease the process of using templates in Typescript. It might be useful for Wingman as well