Add support for CodeLlama prompt

[neofob]

Currently, we do not have prompt style for CodeLlama model(s).

Example:

• Quantized models on TheBloke/CodeLlama-70B-Instruct-GGUF

• Prompt template

  Source: system
  
  {system_message}<step> Source: user
  
  {prompt} <step> Source: assistant

[carlitosmanuelitos]

any success on getting codellama models to deploy within privategpt with file access?

[neofob]

@carlitosmanuelitos : No luck atm yet.

[neofob]

@carlitosmanuelitos : According to Meta webpage, the prompt looks like this

<s>Source: system

System prompt <step> Source: user

First user query <step> Source: assistant
Model response to first query <step> Source: user

Second user query <step> Source: assistant
Destination: user

I tried to build it with the following code. I got the response but after that it went on to infinite about Ethical consideration and guideline...etc.

The patch looks like this:

private_gpt/components/llm/prompt_helper.py

+class CodeLlamaPromptStyle(AbstractPromptStyle):
+    def _messages_to_prompt(self, messages: Sequence[ChatMessage]) -> str:
+        prompt = "<s>"
+        for message in messages:
+            role = message.role.lower()
+            content = message.content or ""
+            if role == "user":
+                prompt += f"Source: {role}\n\n "
+                prompt += f"{content.strip()}"
+                prompt += " <step> "
+            elif role == "assistant":
+                prompt += f"Source: {role}\n"
+                prompt += f"{content.strip()}"
+                prompt += " <step> "
+            elif role == "system":
+                prompt += f"Source: {role}\n\n"
+                prompt += f"{content.strip()}"
+                prompt += " <step> "
+
+        prompt += "Source: assistant\nDestination: user\n\n "
+        return prompt
+
+    def _completion_to_prompt(self, completion: str) -> str:
+        return self._messages_to_prompt(
+            [ChatMessage(content=completion, role=MessageRole.USER)]
+        )

And CodeLlamaPromptStyle is added to the function get_prompt_style.

private_gpt/settings/settings.py

diff --git a/private_gpt/settings/settings.py b/private_gpt/settings/settings.py
index b4fd3ca..d2aec9d 100644
--- a/private_gpt/settings/settings.py
+++ b/private_gpt/settings/settings.py
@@ -113,14 +113,19 @@ class LocalSettings(BaseModel):
     embedding_hf_model_name: str = Field(
         description="Name of the HuggingFace model to use for embeddings"
     )
-    prompt_style: Literal["default", "llama2", "tag", "mistral", "chatml"] = Field(
+    prompt_style: Literal["default", "llama2", "tag", "mistral", "codellama", "chatml"] = Field(
         "llama2",
         description=(
             "The prompt style to use for the chat engine. "
             "If `default` - use the default prompt style from the llama_index. It should look like `role: message`.\n"
             "If `llama2` - use the llama2 prompt style from the llama_index. Based on `<s>`, `[INST]` and `<<SYS>>`.\n"
             "If `tag` - use the `tag` prompt style. It should look like `<|role|>: message`. \n"
-            "If `mistral` - use the `mistral prompt style. It shoudl look like <s>[INST] {System Prompt} [/INST]</s>[INST] { UserInstructions } [/INST]"
+            "If `mistral` - use the `mistral prompt style. It should look like <s>[INST] {System Prompt} [/INST]</s>[INST] { UserInstructions } [/INST]"
+            "`codellama` - use `codellama` prompt style. `Source: system \
+            {system_message}<step> \
+            Source: user \
+            {prompt} <step> \
+            Source: assistant` \n"
             "`llama2` is the historic behaviour. `default` might work better with your custom models."
         ),
     )

I tried another version where there is \n\n for the role assistant but it is the same result. Something is missing here...

The .json files (config, tokenizer..etc.) are from

• https://huggingface.co/codellama/CodeLlama-70b-Instruct-hf

The GGUF files are from:

• https://huggingface.co/TheBloke/CodeLlama-70B-Instruct-GGUF

[neofob]

The current agony:

• https://www.reddit.com/r/LocalLLaMA/comments/1afweyw/quick_headsup_about_using_codellama_70b_and/