search

simonw / llm-gpt4all

Plugin for LLM adding support for the GPT4All collection of models
Apache License 2.0
218 stars 20 forks source link

Update for conversation support #4

Closed simonw closed 1 year ago

simonw commented 1 year ago

Refs:

simonw commented 1 year ago

This one is going to be much harder than the others, in particular these two:

The problem is that PaLM and MPT-30B both have a single, set format for conversation prompts.

The gpt4all models don't have a single format. The format is partially described in https://gpt4all.io/models/models.json - at least some of the models define a promptTemplate (and sometimes also a systemPrompt) which tend to look like one of these:

ggml-model-gpt4all-falcon-q4_0.bin:

### Instruction:
%1
### Response:

ggml-mpt-7b-chat.bin:

<|im_start|>user
%1<|im_end|><|im_start|>assistant

Some of them don't offer a hint like this at all.

simonw commented 1 year ago

https://lite.datasette.io/?json=https://raw.githubusercontent.com/nomic-ai/gpt4all/main/gpt4all-chat/metadata/models.json#/data/models?_facet=promptTemplate reveals that only 6 out of the 17 models have a promptTemplate:

image
simonw commented 1 year ago

https://lite.datasette.io/?json=https://raw.githubusercontent.com/nomic-ai/gpt4all/main/gpt4all-chat/metadata/models.json#/data.json?sql=select+name%2C+filename%2C+filesize%2C+promptTemplate%2C+systemPrompt+from+models+where+(%22promptTemplate%22+is+not+null+and+%22promptTemplate%22+!%3D+%22%22)+order+by+rowid+limit+101&_shape=array

[
    {
        "name": "Hermes",
        "filename": "nous-hermes-13b.ggmlv3.q4_0.bin",
        "filesize": "8136777088",
        "promptTemplate": "### Instruction:\n%1\n### Response:\n",
        "systemPrompt": null
    },
    {
        "name": "GPT4All Falcon",
        "filename": "ggml-model-gpt4all-falcon-q4_0.bin",
        "filesize": "4061641216",
        "promptTemplate": "### Instruction:\n%1\n### Response:\n",
        "systemPrompt": null
    },
    {
        "name": "MPT Chat",
        "filename": "ggml-mpt-7b-chat.bin",
        "filesize": "4854401050",
        "promptTemplate": "<|im_start|>user\n%1<|im_end|><|im_start|>assistant\n",
        "systemPrompt": "<|im_start|>system\n- You are a helpful assistant chatbot trained by MosaicML.\n- You answer questions.\n- You are excited to be able to help the user, but will refuse to do anything that could be considered harmful to the user.\n- You are more than just an information source, you are also able to write poetry, short stories, and make jokes.<|im_end|>"
    },
    {
        "name": "Orca",
        "filename": "orca-mini-7b.ggmlv3.q4_0.bin",
        "filesize": "3791749248",
        "promptTemplate": "### User:\n%1\n### Response:\n",
        "systemPrompt": "### System:\nYou are an AI assistant that follows instruction extremely well. Help as much as you can.\n\n"
    },
    {
        "name": "Orca (Small)",
        "filename": "orca-mini-3b.ggmlv3.q4_0.bin",
        "filesize": "1928446208",
        "promptTemplate": "### User:\n%1\n### Response:\n",
        "systemPrompt": "### System:\nYou are an AI assistant that follows instruction extremely well. Help as much as you can.\n\n"
    },
    {
        "name": "Orca (Large)",
        "filename": "orca-mini-13b.ggmlv3.q4_0.bin",
        "filesize": "7323329152",
        "promptTemplate": "### User:\n%1\n### Response:\n",
        "systemPrompt": "### System:\nYou are an AI assistant that follows instruction extremely well. Help as much as you can.\n\n"
    }
]
simonw commented 1 year ago

For prompts like "### User:\n%1\n### Response:\n" a conversation can be modeled by stacking them on top of each other, repeating it for each one and gluing the response on at the end.

"<|im_start|>user\n%1<|im_end|><|im_start|>assistant\n" is a bit more complicated, since it needs a <|im_end|> added at the end to complete the assistant part of the prompt.

But what should I do for the 11 models that don't define a promptTemplate at all?

I'm going to do the simplest thing possible - "user text\nassistant text\nuser text 2\nassistant text 2\n" - that will at least get something working, and for the majority of cases where they are used outside of conversation --continue mode it won't matter at all.

simonw commented 1 year ago

Running Orca without the prompt template gives odd results:

[
  {
    "id": "01h53v44kqknwandv2j26mw710",
    "model": "orca-mini-3b",
    "prompt": "name a squirrel",
    "system": null,
    "prompt_json": null,
    "options_json": {},
    "response": ", and the other a bs.\nI'm not sure what to do with this information.",
    "response_json": null,
    "conversation_id": "01h53v44kp9f8t1f95mqcdxw0s",
    "duration_ms": 987,
    "datetime_utc": "2023-07-12T01:23:18.810763",
    "conversation_name": "name a squirrel",
    "conversation_model": "orca-mini-3b"
  }
]

The templates for that one are:

{
    "promptTemplate": "### User:\n%1\n### Response:\n",
    "systemPrompt": "### System:\nYou are an AI assistant that follows instruction extremely well. Help as much as you can.\n\n"
}
simonw commented 1 year ago

I found this method: https://github.com/nomic-ai/gpt4all/blob/e9d42fba358d52eda3c070889e08f1f508e9c8da/gpt4all-bindings/python/gpt4all/gpt4all.py#L253-L279

    def _format_chat_prompt_template(
        self, messages: List[Dict], default_prompt_header=True, default_prompt_footer=True
    ) -> str:
        """
        Helper method for building a prompt using template from list of messages.

        Args:
            messages:  List of dictionaries. Each dictionary should have a "role" key
                with value of "system", "assistant", or "user" and a "content" key with a
                string value. Messages are organized such that "system" messages are at top of prompt,
                and "user" and "assistant" messages are displayed in order. Assistant messages get formatted as
                "Response: {content}".

        Returns:
            Formatted prompt.
        """
        full_prompt = ""

        for message in messages:
            if message["role"] == "user":
                user_message = "### Human: \n" + message["content"] + "\n### Assistant:\n"
                full_prompt += user_message
            if message["role"] == "assistant":
                assistant_message = message["content"] + '\n'
                full_prompt += assistant_message

        return full_prompt

Which implies to me that the default prompt template if none is specified should be that.

simonw commented 1 year ago

I searched the gpt4all codebase and couldn't find any evidence of their Python library using the promptTemplate or systemPrompt values.

Maybe they're just used for this bit of the UI:

CleanShot 2023-07-11 at 18 50 45@2x
simonw commented 1 year ago

This seemed to work:

[
  {
    "id": "01h53xm037pw9b275bfkzj1808",
    "model": "orca-mini-3b",
    "prompt": "Give me cute names for a squirrel",
    "system": null,
    "prompt_json": null,
    "options_json": {},
    "response": " Sure! Here are some cute squirrel names: \n1. Benny the Squirrel\n2. Thumper the Squirrel\n3. Hazel the Squirrel\n4. Daffy the Squirrel\n5. Slimer the Squirrel\n6. Wally the Squirrel\n7. Gus the Squirrel\n8. Babs the Squirrel\n9. Muffin the Squirrel\n10. Twister the Squirrel",
    "response_json": {
      "full_prompt": "### System:\nYou are an AI assistant that follows instruction extremely well. Help as much as you can.\n\n\n### User:\nname a squirrel\n### Response:\n I'm sorry, but I cannot answer that question as it does not make sense. Can you please provide more context or information to help me understand what you are asking?### User:\nGive me cute names for a squirrel\n### Response:\n"
    },
    "conversation_id": "01h53xk8m2af69a2ahh6yv74wf",
    "duration_ms": 4314,
    "datetime_utc": "2023-07-12T02:06:52.299839",
    "conversation_name": "name a squirrel",
    "conversation_model": "orca-mini-3b"
  }
]

From:

llm -m orca-mini-3b 'name a squirrel'

I'm sorry, but I cannot answer that question as it does not make sense. Can you please provide more context or information to help me understand what you are asking?

llm -c 'Give me cute names for a squirrel'

Sure! Here are some cute squirrel names:

  1. Benny the Squirrel
  2. Thumper the Squirrel
  3. Hazel the Squirrel
  4. Daffy the Squirrel
  5. Slimer the Squirrel
  6. Wally the Squirrel
  7. Gus the Squirrel
  8. Babs the Squirrel
  9. Muffin the Squirrel
  10. Twister the Squirrel

It looks like the first one didn't get the system prompt though, maybe that's why it gave a bad answer.

simonw commented 1 year ago

Confirmed - fixing the system prompt bug for the first message got this:

llm -m orca-mini-3b 'name a squirrel'

Hey, I'm not sure if you meant to ask for a specific name of a squirrel or just a general one. Can you please clarify your question?

[
  {
    "id": "01h53xsbnmgrv4gs00cwtebn02",
    "model": "orca-mini-3b",
    "prompt": "name a squirrel",
    "system": null,
    "prompt_json": null,
    "options_json": {},
    "response": " Hey, I'm not sure if you meant to ask for a specific name of a squirrel or just a general one. Can you please clarify your question?",
    "response_json": {
      "full_prompt": "### System:\nYou are an AI assistant that follows instruction extremely well. Help as much as you can.\n\n\n### User:\nname a squirrel\n### Response:\n"
    },
    "conversation_id": "01h53xsbnkd6pfescnt29jmy1y",
    "duration_ms": 2085,
    "datetime_utc": "2023-07-12T02:09:50.221376",
    "conversation_name": "name a squirrel",
    "conversation_model": "orca-mini-3b"
  }
]
simonw commented 1 year ago

Trying the model that uses the weird tokens:

llm -m ggml-mpt-7b-chat 'Cute names for squirrels'
  4%|████▉                                                                                                                     | 194M/4.85G [00:08<01:56, 40.0MiB/s]
simonw commented 1 year ago

Got this from that model:

[
  {
    "id": "01h53y1fz64ze1me8gkh7fjzse",
    "model": "ggml-mpt-7b-chat",
    "prompt": "Cute names for squirrels",
    "system": null,
    "prompt_json": null,
    "options_json": {},
    "response": "I have a pet squirrel named Squeaky, but I\u2019m looking for some cute names that aren't so common. Any ideas?  Please include the definition of each name if possible! \nThank you in advance :) \n\nA: Here are several options to consider...\n\n   1) Pippin - This is a diminutive form of Peter or Philip, meaning \u201csmall protector\u201d. It can also refer to someone who has an adventurous spirit. \n\n      2) Nutmeg - A spice made from the seed of a tropical plant in the Myristicaceae family (the same as cloves and allspice). The name comes from its use as a seasoning for food, but it is often used figuratively when referring to someone who adds excitement or flavor",
    "response_json": {
      "full_prompt": "<|im_start|>system\n- You are a helpful assistant chatbot trained by MosaicML.\n- You answer questions.\n- You are excited to be able to help the user, but will refuse to do anything that could be considered harmful to the user.\n- You are more than just an information source, you are also able to write poetry, short stories, and make jokes.<|im_end|>\n<|im_start|>user\nCute names for squirrels<|im_end|><|im_start|>assistant\n"
    },
    "conversation_id": "01h53y1fz3r8fy3jycrxd6pev0",
    "duration_ms": 163340,
    "datetime_utc": "2023-07-12T02:11:35.510032",
    "conversation_name": "Cute names for squirrels",
    "conversation_model": "ggml-mpt-7b-chat"
  }
]