Open synw opened 1 year ago
I totally agree. This is another one of the things handled by the lib I use for request/response formatting. Specifically, this happens here:
https://github.com/transitive-bullshit/chatgpt-api/blob/main/src/chatgpt-api.ts#L395
Maybe it's time to roll my own solution for this, so that a prompt formatting can be explicitly defined. I'd like to implement this, but need some time to think about how best to do so.
I studied this library recently when trying to implement OpenAi support in my server project. It is easy to replace: example code:
Nice, seems easy enough. I should have time to get this done this evening.
How would you expect to define a template? I think I'd probably surface this as an item in the wingman settings panel as e.g. wingman.openai.template
, as well as allow it to be overridden by any given command. Does that seem okay to you?
yes it looks good. I would define a template in a very simple maner. An Alpaca one:
const template = `{system}
### Instruction: fix this invalid json:
'''json
{prompt}
'''
### Response: (answer in valid json)
`
const system = "You are a javascript code specialist. Below is an instruction that describes a task. Write a response that appropriately completes the request";
const _prompt = '{"a": 1,}';
const finalPrompt = template.replace("{system}", system).replace("{prompt}", _prompt)
where {system}
is actually covered by systemMessageTemplate
in Wingman
Note about handling the sse request: the eventsource-parser lib is not really necessary: we can handle this manually using a regex, this is what I did in my frontend: example
About templates I just started a little library to manage them more efficiently an use prebuilt template formats: https://github.com/synw/modprompt
I got tired to copy and change text to different template formats depending on the models, so I wrote this to ease the process of using templates in Typescript. It might be useful for Wingman as well
A nice feature to have would be to be able to select a prompt format. The prompt templates are different for each model family. I see that you use the Alpaca format (
### Instruction: ... {prompt}\n\n### Response:
). For example it could be nice to be able to change the prompt template to for example the Llama 2 format:or Orca 2:
[Edit] Example use case: I would like to use https://huggingface.co/TheBloke/LosslessMegaCoder-Llama2-7B-Mini-GGML that looks good on coding tasks, but it's template format is ChatMl: