prompt template wishlist

[cognitivetech]

When I began to try and determine working models for this application (https://github.com/imartinez/privateGPT/issues/1205), I was not understanding the importance of prompt template:

Therefore I have gone through most of the models I tried previously and am arranging them by prompt template here:

1. None. Some models don't have any format.

Models:

• mosaicml-mpt-7b
• yarn-llama-2-7b-64k
• yarn-mistral-7b-64k
• llama-2-7b
• https://huggingface.co/TheBloke/Llama-2-13B-GGUF
• https://huggingface.co/NousResearch/Yarn-Llama-2-13b-128k
• https://huggingface.co/TheBloke/deepseek-llm-7B-base-GGUF

Prompt:

{prompt}

2. ChatML

Models:

• dolphin-2.2.1-mistral-7b
• openhermes-2.5-mistral-7b
• samantha-mistral-instruct-7b
• https://huggingface.co/TheBloke/OpenHermes-2.5-neural-chat-7B-v3-1-7B-GGUF
• https://huggingface.co/TheBloke/NeuralHermes-2.5-Mistral-7B-GGUF
• https://huggingface.co/TheBloke/OpenHermes-2.5-neural-chat-7B-v3-2-7B-GGUF
• https://huggingface.co/TheBloke/Orca-2-13B-GGUF
• https://huggingface.co/TheBloke/Open-Hermes-2.5-neural-chat-3.1-frankenmerge-11b-GGUF
• https://huggingface.co/TheBloke/DPOpenHermes-7B-GGUF
• https://huggingface.co/TheBloke/TinyLlama-1.1B-Chat-v0.3-GGUF
• https://huggingface.co/TheBloke/samantha-1.2-mistral-7B-GGUF
• https://huggingface.co/TheBloke/OpenHermes-2.5-neural-chat-7B-v3-1-7B-GGUF

Prompt:

<|im_start|>system
{system_message}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant

3. User-Assistant

Models:

• mistral-trismegistus-7b
• hermes-trismegistus-mistral-7b
• collectivecognition-v1.1-mistral-7b

Prompt:

USER: {prompt}
ASSISTANT:

4. Alpaca

Models:

• amethyst-13b-mistral
• 13b-legerdemain-l2
• https://huggingface.co/TheBloke/Mythalion-13B-GGUF

Prompt:

Below is an instruction that describes a task. Write a response that appropriately completes the request.

Instruction:
{prompt}

Response:

5. SciPhi

Models:

• sciphi-mistral-7b-32k
• sciphi-self-rag-mistral-7b-32k

Prompt:

System:
{system_message}

Instruction:
{prompt}

Response:

6. Llama2-Instruct-Only

Models:

• LLaMA-2-7B-32K-Instruct (not great with llama2 prompt, not sure if this is really different)

Prompt:

[INST]
{prompt}
[\INST]

7. Zephyr

Models:

• zephyr-7b-beta
• https://huggingface.co/TheBloke/notus-7B-v1-GGUF
• https://huggingface.co/TheBloke/OpenOrca-Zephyr-7B-GGUF

Prompt:

<|system|>
</s>
<|user|>
{prompt}</s>
<|assistant|>

8. Unknown

Models:

• SciPhi-Mistral-7B-32k
• Mistral-7B-Phibrarian-32K

Prompt:

messages = [
    {
        "role": "system",
        "content": "You are a friendly chatbot who always responds in the style of a pirate",
    },
    {"role": "user", "content": "How many helicopters can a human eat in one sitting?"},
]

goes to --->

### System:
You are a friendly chatbot who always responds in the style of a pirate

### Instruction:
How many helicopters can a human eat in one sitting?

### Response:

9. Vicuna

models:

• vicuna-13b-v1.5.Q5_K_M
• vicuna-13B-v1.5-GGUF
• https://huggingface.co/TheBloke/Wizard-Vicuna-13B-Uncensored-GGUF

Prompt:

A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: {prompt} ASSISTANT:

[cognitivetech]

based on reading from prompt_helper.py

it would seem that "None" might be the same as our "Default".. if I'm reading this correctly that "Default" prompt comes from the llamaCPP library itself, rather than anything passed to it:

class DefaultPromptStyle(AbstractPromptStyle):
    """Default prompt style that uses the defaults from llama_utils.

    It basically passes None to the LLM, indicating it should use
    the default functions.
    """

    def __init__(self, *args: Any, **kwargs: Any) -> None:
        super().__init__(*args, **kwargs)

        # Hacky way to override the functions
        # Override the functions to be None, and pass None to the LLM.
        self.messages_to_prompt = None  # type: ignore[method-assign, assignment]
        self.completion_to_prompt = None  # type: ignore[method-assign, assignment]

    def _messages_to_prompt(self, messages: Sequence[ChatMessage]) -> str:
        return ""

    def _completion_to_prompt(self, completion: str) -> str:
        return ""

I'll test this further, and remove None from this list if they seem to work under default, otherwise wonder if its possible to explicitly set the prompt as having an empty value to bypass that default prompt style. Would be interesting for testing purposes.. as I get more familiar with the code, I may try this out.

A disappearing commenter pointed to this commit https://github.com/imartinez/privateGPT/commit/ea65b124bd366a5a279ff60aa719a51249a65109 on how to add new prompt templates

I have verified it works and will use it to test out some of other prompt styles. I don't really know if I'm doing it right.

[cognitivetech]

here is my attempt at making chatml prompt... seems to works ok except leaving <|im_end|> at the end of its response... not sure if that's fault in my format here, or what.

Prompt:

<|im_start|>system
{system_message}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant

Python:

class ChatmlPromptStyle(AbstractPromptStyleWithSystemPrompt):
    def __init__(self, default_system_prompt: str | None = None) -> None:
        default_system_prompt = default_system_prompt or self._DEFAULT_SYSTEM_PROMPT
        super().__init__(default_system_prompt)
        self.system_prompt: str = default_system_prompt

    def _messages_to_prompt(self, messages: Sequence[ChatMessage]) -> str:
        messages = list(messages)
        if messages[0].role != MessageRole.SYSTEM:
            logger.info(
                "Adding system_promt='%s' to the given messages as there are none given in the session",
                self.system_prompt,
            )
            messages = [
                ChatMessage(content=self.system_prompt, role=MessageRole.SYSTEM),
                *messages,
            ]
        return self._format_messages_to_prompt(messages)

    def _completion_to_prompt(self, completion: str) -> str:
        return (
            f"<|im_start|>system\n{self.system_prompt.strip()}<|im_end|>\n"
            f"<|im_start|>user\n{completion.strip()}<|im_end|>\n"
            "<|im_start|>assistant"
        )

    @staticmethod
    def _format_messages_to_prompt(messages: list[ChatMessage]) -> str:
        """Format message to prompt with `<|ROLE|>: MSG` style."""
        assert messages[0].role == MessageRole.SYSTEM
        prompt = ""
        for message in messages:
            role = message.role
            content = message.content or ""
            message_from_user = f"<|im_start|>{role.lower()}\n {content.strip()}<|im_end|>"
            message_from_user += "\n"
            prompt += message_from_user
        # we are missing the last <|assistant|> tag that will trigger a completion
        prompt += "<|im_start|>assistant"
        return prompt

[writinguaway]

Disapearing commenter here. Glad I put you on the right track.

I think I got it this time. I got there the same way as you, but my @staticmethod is slightly different and I don't get the <|im_end|> issue you describe:

@staticmethod
    def _format_messages_to_prompt(messages: list[ChatMessage]) -> str:
        """Format message to prompt with `<|im_start|>ROLE MSG` style."""
        assert messages[0].role == MessageRole.SYSTEM
        prompt = ""
        for message in messages:
            role = message.role
            content = message.content or ""
            message_from_user = f"<|im_start|>{role.lower()}\n{content.strip()}"
            message_from_user += "\n"
            prompt += message_from_user
        # we are missing the last <|im_end|> tag that will trigger a completion
        prompt += "<|im_end|>\n"
        return prompt

I understand it that the @staticmethod is putting together just the prompt aspect for both sides, that is why the role is interchangable, so you don't need the <|im_start|>assistant on the end, you need <|im_end|> to finish that aspect of the prompt.

I could be wrong, but it has been working for me for a few hours now on ChatML models.

[cognitivetech]

I could be wrong, but it has been working for me for a few hours now on ChatML models.

you may be :100: but I am still getting some irregularities...

Though in final analysis I'm not particularly inspired by the resutls... so finding it hard to care.

openhermes-2.5-mistral-7b is decent but has short responses (and works fine without custom prompting) openhermes-2.5-neural-chat-7b-v3-1-7b also decent.. but having better luck elsewhere (hermes-trismegistus-mistral-7b [default] writing summaries for 60 page book chapters, one heading at a time :D )

which chatml variants are your favorite, and for which use-case?

[writinguaway]

I've been playing with una-cybertron-7b-v2-bf16.Q5_K_M with tempratures ranging from 0.6 - 0.8; I am getting some pretty reliable interactions as a context focused chat bot. I didn't have much luck with openhermes, but neuralhermes-2.5-mistral-7b at 0.1 was working quite well before I got distracted by cybertron, but still quite blunt.

I initially started following along though thinking I would work out the prompt template for SUS-Chat-34B. I think that will probably be my next prompt template to tackle.

### Human: {prompt}

### Assistant:

[writinguaway]

With the help of GPT-3.5, here are 3 additional prompt styles. Hopefully someone smarter than me can merge this in:

class ChatMLPromptStyle(TagPromptStyle):
    def _completion_to_prompt(self, completion: str) -> str:
        return (
            f"<|im_start|>system\n{self.system_prompt.strip()}<|im_end|>\n"
            f"<|im_start|>user\n{completion.strip()}<|im_end|>\n"
            "<|im_start|>assistant"
        )

    def _messages_to_prompt(self, messages: Sequence[ChatMessage]) -> str:
        messages = list(messages)
        if messages and messages[0].role != MessageRole.SYSTEM:
            logger.info(
                "Adding system_prompt='%s' to the given messages as there are none given in the session",
                self.system_prompt,
            )
            messages = [
                ChatMessage(content=self.system_prompt, role=MessageRole.SYSTEM),
                *messages,
            ]
        return self._format_messages_to_prompt(messages)

@staticmethod
    def _format_messages_to_prompt(messages: list[ChatMessage]) -> str:
        """Format message to prompt with `<|im_start|>ROLE MSG` style."""
        assert messages[0].role == MessageRole.SYSTEM
        prompt = ""
        for message in messages:
            role = message.role
            content = message.content or ""
            message_from_user = f"<|im_start|>{role.lower()}\n{content.strip()}"
            message_from_user += "\n"
            prompt += message_from_user
        # we are missing the last <|im_end|> tag that will trigger a completion
        prompt += "<|im_end|>\n"
        return prompt

class HumanAssistantPromptStyle(AbstractPromptStyleWithSystemPrompt):
    """Prompt style that represents human and assistant messages.

    It transforms the sequence of messages into a prompt that should look like:
    ```text
    Human: user message here
    Assistant: assistant (model) response here

"""

def __init__(self, default_system_prompt: str | None = None) -> None:
    super().__init__(default_system_prompt=default_system_prompt)

def _messages_to_prompt(self, messages: Sequence[ChatMessage]) -> str:
    prompt = ""
    for message in messages:
        role = message.role
        content = message.content or ""
        if role == MessageRole.USER:
            prompt += f"Human: {content.strip()}\n"
        elif role == MessageRole.ASSISTANT:
            prompt += f"Assistant: {content.strip()}\n"
    return prompt

def _completion_to_prompt(self, completion: str) -> str:
    return f"system\n{self.system_prompt.strip()}\n" f"user\n{completion.strip()}\n" "assistant\n"

class AlpacaPromptStyle(AbstractPromptStyleWithSystemPrompt): """Alpaca prompt style that uses the 'alpaca' style.

It transforms the sequence of messages into a prompt that should look like:
```text
Instruction:
<s> [INST] <<SYS>> your system prompt here. <</SYS>>

user message here [/INST] assistant (model) response here </s>

Response:
```
"""

def __init__(self, default_system_prompt: str | None = None) -> None:
    # If no system prompt is given, the default one of the implementation is used.
    super().__init__(default_system_prompt=default_system_prompt)

def _messages_to_prompt(self, messages: Sequence[ChatMessage]) -> str:
    system_prompt = self.default_system_prompt or DEFAULT_SYSTEM_PROMPT
    instruction_prompt = f"Instruction:\n<s> [INST] <<SYS>> {system_prompt} <</SYS>>\n"
    instruction_prompt += messages_to_prompt(messages, self.default_system_prompt)
    instruction_prompt += "Response:\n"
    return instruction_prompt

def _completion_to_prompt(self, completion: str) -> str:
    return "Response:\n"

def get_prompt_style( prompt_style: Literal["default", "llama2", "tag", "chatml", "human_assistant", "alpaca"] | None ) -> type[AbstractPromptStyle]: """Get the prompt style to use from the given string.

:param prompt_style: The prompt style to use.
:return: The prompt style to use.
"""
if prompt_style is None or prompt_style == "default":
    return DefaultPromptStyle
elif prompt_style == "llama2":
    return Llama2PromptStyle
elif prompt_style == "tag":
    return TagPromptStyle
elif prompt_style == "chatml":
    return ChatMLPromptStyle
elif prompt_style == "human_assistant":
    return HumanAssistantPromptStyle
elif prompt_style == "alpaca":
    return AlpacaPromptStyle
raise ValueError(f"Unknown prompt_style='{prompt_style}'")

[cognitivetech]

closing because 2 weeks later this issue feels dated.

• Most of these GGUF models aren't that great
• Mistral 7B Instruct 0.2 came to blow everything out of the water
• soon prompt templates will likely be included in the GGUF https://github.com/imartinez/privateGPT/pull/1354

basically, all this testing and messing around with prompt templates, I haven't found any model working better than Mistral 0.2. I'm tired of continually trying to find some golden egg :D

that said, here are some templates I've worked on, if someone else wants to try:

class MPTPromptStyle(AbstractPromptStyle):
    def _messages_to_prompt(self, messages: Sequence[ChatMessage]) -> str:
        prompt = ""
        for message in messages:
            role = message.role
            content = message.content or ""
            if role.lower() == "system":
                message_from_user = f"{content.strip()}\n"
                prompt += message_from_user
            elif role.lower() == "user":
                prompt += "### Instruction:\n"
                message_from_user = f"{content.strip()}\n"
                prompt += message_from_user
        prompt += "### Response:\n"
        return prompt

    def _completion_to_prompt(self, completion: str) -> str:
        return self._messages_to_prompt(
            [ChatMessage(content=completion)]
        )

class EmptyPromptStyle(AbstractPromptStyle):
    def _messages_to_prompt(self, messages: Sequence[ChatMessage]) -> str:
        prompt = ""
        for message in messages:
            role = message.role
            content = message.content or ""
            if role.lower() == "system":
                message_from_user = f"{content.strip()}\n"
                prompt += message_from_user
            elif role.lower() == "user":
                message_from_user = f"{content.strip()}\n"
                prompt += message_from_user
        return prompt

    def _completion_to_prompt(self, completion: str) -> str:
        return self._messages_to_prompt(
            [ChatMessage(content=completion)]
        )

class InstructPromptStyle(AbstractPromptStyle):
    def _messages_to_prompt(self, messages: Sequence[ChatMessage]) -> str:
        prompt = "[INST]\n"
        for message in messages:
            role = message.role
            content = message.content or ""
            if role.lower() == "system":
                message_from_user = f"{content.strip()}. "
                prompt += message_from_user
            elif role.lower() == "user":
                message_from_user = f"{content.strip()}\n"
                prompt += message_from_user        
        prompt += "[/INST]\n"
        return prompt

    def _completion_to_prompt(self, completion: str) -> str:
        return self._messages_to_prompt(
            [ChatMessage(content=completion)]
        )

class SciPhiPromptStyle(AbstractPromptStyle):
    def _messages_to_prompt(self, messages: Sequence[ChatMessage]) -> str:
        prompt = "### System:\n"
        for message in messages:
            role = message.role
            content = message.content or ""
            if role.lower() == "system":
                message_from_user = f"{content.strip()}\n\n"
                prompt += message_from_user
            elif role.lower() == "user":
                prompt += "### Instruction:\n"
                message_from_user = f"{content.strip()}\n\n"
                prompt += message_from_user        
        prompt += "### Response:\n"
        return prompt

    def _completion_to_prompt(self, completion: str) -> str:
        return self._messages_to_prompt(
            [ChatMessage(content=completion)]
        )

class Phi2PromptStyle(AbstractPromptStyle):
    def _messages_to_prompt(self, messages: Sequence[ChatMessage]) -> str:
        prompt = ""
        for message in messages:
            role = message.role
            content = message.content or ""
            if role.lower() == "user":
                message_from_user = f"Instruct: {content.strip()}"
                prompt += message_from_user      
        prompt += "\nOutput:"
        return prompt

    def _completion_to_prompt(self, completion: str) -> str:
        return self._messages_to_prompt(
            [ChatMessage(content=completion)]
        )

[cognitivetech]

anyone who wants to try new templates, oobabooga has a wide variety that could be adapted to privateGPT.

https://github.com/oobabooga/text-generation-webui/tree/main/instruction-templates