Model config system_prompt not being used in chat

[mattlindsey]

Using the admin page, I added the system_prompt Please call me Jeremy to the default phi3:latest chat model, but when chatting I don't see it being used.

In the api call for Hi. Do you know my name? I just see the message below (from Message.summary_chat method). Shouldn't it use use or add a role: system message?

{
  "model": "phi3:latest",
  "messages": [
    {
      "role": "system",
      "content": "You are a summarization AI. You'll never answer a user's question directly, but instead summarize\nthe user's request into a single short sentence of four words or less. Always start your answer\nwith an emoji relevant to the summary\",\n"
    },
    {
      "role": "user",
      "content": "Who is the president of Gabon?"
    },
    {
      "role": "assistant",
      "content": "🇬🇦 President of Gabon"
    },
    {
      "role": "user",
      "content": "Who is Julien Chaumond?"
    },
    {
      "role": "assistant",
      "content": "🧑 Julien Chaumond"
    },
    {
      "role": "user",
      "content": "what is 1 + 1?"
    },
    {
      "role": "assistant",
      "content": "🔢 Simple math operation"
    },
    {
      "role": "user",
      "content": "What are the latest news?"
    },
    {
      "role": "assistant",
      "content": "📰 Latest news"
    },
    {
      "role": "user",
      "content": "How to make a great cheesecake?"
    },
    {
      "role": "assistant",
      "content": "🍰 Cheesecake recipe"
    },
    {
      "role": "user",
      "content": "what is your favorite movie? do a short answer."
    },
    {
      "role": "assistant",
      "content": "🎥 Favorite movie"
    },
    {
      "role": "user",
      "content": "Explain the concept of artificial intelligence in one sentence"
    },
    {
      "role": "assistant",
      "content": "🤖 AI definition"
    },
    {
      "role": "user",
      "content": "Draw a cute cat"
    },
    {
      "role": "assistant",
      "content": "🐱 Cute cat drawing"
    },
    {
      "role": "user",
      "content": "Hi. Do you know my name?"
    }
  ]
}

Response body:

🙋‍♂️ I don't have access to personal data unless it's shared with me in the context of our conversation, and even then, privacy is paramount. My primary function is to assist without retaining identifying information about individuals beyond this interaction for their safety and confidentiality.

[nickthecook]

Yes, the system prompt is in the schema but isn't actually used.

For each chat message a user submits, two calls to an LLM are made: the summarization call, which Archyve uses to set the conversation title, and the actual chat request.

The request you found is the summarization request, which doesn't include any augmentation, and I'm not sure if it should include the system prompt or not. The chat request, however, definitely should. This is an issue right now, definitely a basic feature we want to have.

[nickthecook]

Hey @mattlindsey, where I would like to go with the system prompt is have an Archyve-specific system prompt that makes the model aware that it is playing the role of Archyve, with access to certain collections and documents. I'd also like to implement tool use to have the model able to control the search process itself.

The advantages of this include being able to ask questions about documents and collections, like "What documents do you have access to?" or say things like "In the refrigerator manual, how do I ...?" and have the LLM make the decision to only search that document.

This is a little ways off yet, but as a first step, the system prompt could be used to describe the role of Archyve so that the LLM responses aren't naive chat, but that of a bot in this role. It will be somewhat limited without the tool use piece, but I'm curious how it would turn out, and if just setting the correct system prompt alone is enough to get a better experience out of Archyve.

[mattlindsey]

Hi @nickthecook. I can do a PR to enable the system prompt and add a prompy to try out. Where should the text of the prompt go? Rather than do something like the Message.summary_chat method above or system_prompt in the model_configs table, maybe it should go into the config table as chat_system_prompt?
Implementing some tools sound interesting. Which tools? Custom ones, or have you seen something useful implemented already? Both Ollama and Openai support tools, which is nice.

[mattlindsey]

Also, is the purpose of the chat feature in the archyve UI to help a person manage the documents better? Or is it also meant to let a person do actual research on the documents too? The arcyve api doesn't have a chat, does it?

P.S. Oh, I see the api has chat (below). Would that use the chat system prompt too?

/v1/chat?prompt=<prompt>&model=<model>&augment=<true|false>&collections=<collection_ids>
send a chat prompt and return the response

It feels like maybe the system prompt should be visible/editable right on the Chat screen in Archyve.

[oxaronick]

I see the chat as two things, in this order:

1. a way to test Archyve's RAG in a chat scenario
2. a functional chat, for people who aren't already running something else

Seems like overriding the system prompt for a Conversation right in the chat is in the spirit of number 1. I'd use that when it came time to draft a default system prompt to give Archyve its identity.

Actually, it seems like having a Setting like system_prompt that is "global", and then overriding that Setting per Conversation would be a good way to support this. You can associate a Setting with any model by passing a target param to Setting#set or any of its other public methods. If target is nil, you're dealing with "global" Settings. (I plan to have some method in Setting, like maybe find that will check the given target and then, if no Setting is found for that target, check global Settings, which would be useful here.)

The chat API could eventually support the system prompt param as well, and set that for the Conversation when creating it.

What do you think?

[mattlindsey]

Sounds good.
So tasks for this issue would be:

• [ ] Add system_prompt to Settings that is used globally in chat (but not summary call). Seed a basic default in Settings.
• [ ] Ensure system_prompt can be over-ridden for a specific model using target param. Maybe add Setting.find method.
• [ ] Add 'System Instructions' field to chat Converstion UI that overrides system_prompt. Maybe add a note on screen saying that so user knows what's happening, or if a system_prompt for the specific model exists in Settings, should it automatically appear in 'System Instructions' when a new Conversation is started? Maybe that should happen with the global one too?
• [ ] Remove system_prompt from model_configs table
• [ ] Display and allow editing of global system_prompt on /settings page

[nickthecook]

Ensure system_prompt can be over-ridden for a specific model using target param.

I had been thinking of overriding it for a Conversation rather than a ModelConfig. If it's a setting on the ModelConfig, I feel like showing that within the Conversation view wouldn't necessarily make sense. We would need to flesh out the model settings quite a bit more (I'm working on this now, including UI to show the models that got pulled from a given ModelServer).

If it's a global prompt and optional Conversation-specific prompts as a first step, I think that makes sense. That would let the user test out multiple system prompts, and we can always add the same Setting to ModelConfig later if it makes sense.

What do you think?

...should it automatically appear in 'System Instructions' when a new Conversation is started?

Sounds like a good idea to put the global one in the field by default; then the user can just edit that one to test out a new prompt.

I worry a little about cluttering up the Conversation view. Would you hide this field and require the user to click to display it?

In general, I think we could do more with system prompt, but this is a good way to get started!