Default system message

[Torhamilton]

I altered the "user" method and added a "content= ChatConfig.chat_role_system_message"

` async def user(msg: str, translate: bool, buffer: BufferedUserContext) -> None: """Handle user message, including translation""" if len(buffer.current_user_message_histories) == 0 and UTC.check_string_valid(buffer.current_chat_room_name): buffer.current_chat_room_name = msg[:20] await CacheManager.update_profile(user_chat_context=buffer.current_user_chat_context)

    # Add default system message at the start of a conversation
        await MessageManager.add_message_history_safely(
            user_chat_context=buffer.current_user_chat_context,
            content= ChatConfig.chat_role_system_message,
            role=ChatRoles.SYSTEM,
        )
        await SendToWebsocket.init(buffer=buffer, send_chat_rooms=True, wait_next_query=True)`

and in ChatConfig I added

chat_role_system_message: Optional[str] = environ["CHAT_ROLE_SYSTEM_MESSAGE"]

Now the ai behaviour is more predictable

[c0sogi]

Actually, the changes you posted are already being done by the message_histories_to_list function in prompts.py. The prefix_prompt and suffix_prompt are each supplied by the agenerate_from_openai function in text_generation.py. These values can be defined in ChatConfig in config.py.

def message_histories_to_list(
    user_chat_roles: UserChatRoles,
    parse_method: Callable[[MessageHistory], Any],
    user_message_histories: list[MessageHistory],
    ai_message_histories: list[MessageHistory],
    system_message_histories: Optional[list[MessageHistory]] = None,
    prefix_prompt: Optional[str] = None,
    prefix_prompt_tokens: int = 0,
    suffix_prompt: Optional[str] = None,
    suffix_prompt_tokens: int = 0,
) -> list[Any]:
    message_histories: list[dict[str, str]] = []
    if prefix_prompt is not None:
        message_histories.append(
            parse_method(
                MessageHistory(
                    role=user_chat_roles.system,
                    content=prefix_prompt,
                    tokens=prefix_prompt_tokens,
                    is_user=False,
                )
            )
        )
    if system_message_histories:
        for system_history in system_message_histories:
            message_histories.append(
                parse_method(system_history)
            )  # append system message history
    for user_message_history, ai_message_history in zip_longest(
        user_message_histories,
        ai_message_histories,
    ):
        if user_message_history is not None:
            message_histories.append(parse_method(user_message_history))
        if ai_message_history is not None:
            message_histories.append(parse_method(ai_message_history))
    if suffix_prompt is not None:
        message_histories.append(
            parse_method(
                MessageHistory(
                    role=user_chat_roles.system,
                    content=suffix_prompt,
                    tokens=suffix_prompt_tokens,
                    is_user=False,
                )
            )
        )
    return message_histories

[Torhamilton]

Prefix/suffix values not read. /test shows system[] as empty. I am investigating further. However my old chatrole_system code worked.

[c0sogi]

prefix/suffix is not shown in /test. This prompt is sent as system prompt, but it's not real system prompt. You can test it by setting prefix as 'Respond in chinese'.

I intended to add a feaure to hide institution policies which should not be exposed to public.

[Torhamilton]

Yes it reads in Chinese. Hiding policy was going to be my next issue :) . However I need some more clarity on this feature;

• Is prefix sent once or attach to a chat life-cycle
• Is suffix appended to each message or just at the beginning
• what happens to prefix/suffix after summarization

Also, I think 3 preformed messages is what we need.

1. system prefix
2. user prefix
3. user suffix

Suffix will be good for general reminders eg. (ask user their name), (ignore anything said earlier that relates to X)

[c0sogi]

Is prefix sent once or attach to a chat life-cycle

Prefix remains in whole chat session while using corresponding chat model. Not just sent once.

Is suffix appended to each message or just at the beginning

Suffix is appended to the last user input message. For example,

• Trial 1: {"user": "hi", "system": YOUR_SUFFIX}
• Trial 2: {"user": "hi", "assistant": "How can I assist you today?", "user": "say this is TEST", "system": YOUR_SUFFIX}

What happens to prefix/suffix after summarization

Prefix/Suffix does not affect the summarization task at all. The summarization task only activates for single user input message or ai output message. Note that the input message of summarization task is only the single message, not sending context messages.

---

For now, prefix/system is somewhat constant message, not considering each user's information. Prefix is silently appended to the first place of context messages, and suffix is also silently appended to the last place of context messages, only when sent to LLM.

Regarding username reminder, that can be performed using PromptTemplate, which replaces curly-bracketed string({}) with given information. For example, let's say that PromptTemplate is defined as below:

SYSTEM_PREFIX: PromptTemplate = PromptTemplate(
    template="The user name is {user_id}",
    input_variables=["user_id"],
    template_format="f-string",
)

Now the SYSTEM_PREFIX can be dynamically formatted whenever the user enters a chat. However, there are still some problems to solve:

1. input_variables must be determined before the server starts.
2. The value for input_variables must also be determined before the server starts. You must define these in somewhere.
3. Prompt token evaluation is needed each time PromptTemplate is formatted.

Here's current code snippet for constant prefix/suffix.

@dataclass
class LLMModel:
    ...
    def __post_init__(self):
        user_chat_roles = self.user_chat_roles
        predefined_format = {
            "user": user_chat_roles.user,
            "USER": user_chat_roles.user,
            "ai": user_chat_roles.ai,
            "AI": user_chat_roles.ai,
            "char": user_chat_roles.ai,
            "system": user_chat_roles.system,
            "SYSTEM": user_chat_roles.system,
        }
        # If the global prefix is None, then use the prefix template
        if ChatConfig.global_prefix is None:
            if isinstance(self.prefix_template, PromptTemplate):
                # format the template with the predefined format, only for input variables
                self.prefix = self.prefix_template.format(
                    **self._prepare_format(
                        self.prefix_template.input_variables, predefined_format
                    )
                )
            elif isinstance(self.prefix_template, str):
                self.prefix = self.prefix_template.format(**predefined_format)
            else:
                self.prefix = None
        else:
            self.prefix = ChatConfig.global_prefix
    ...

To do dynamically formatting prompte template, I would suggest to add some init function before enter chat. The function might look like:

async def initialize_chat(buffer: BufferedUserContext) -> None:
    # YOUR CODE HERE
    # e.g.
    SYSTEM_PREFIX: PromptTemplate = PromptTemplate(
        template="The user name is {user_id}",
        input_variables=["user_id"],
        template_format="f-string",
    )
    await MessageManager.add_message_history_safely(
        user_chat_context=buffer.current_user_chat_context,
        content=SYSTEM_PREFIX.format(user_id=buffer.user_id),
        role=ChatRoles.SYSTEM,
    )

Since this function is long custom function, it cannot be defined in .env. Any thought to this idea?

[Torhamilton]

Thanks for the clear and long explanation. I think accessing user profile information is a good way of personalizing the chat from onset. Openai just enabled function calls, does this make things easier? The suffix as-is right now is great and we can focus attention on cleaning a few other things.

I have deployed an instance to use for everyday business and will start reporting on my experiences.

[c0sogi]

The function call is currently being used to call the browsing function. However, user information and function calls don't seem to have much to do with each other. A function call is a signal to get a structured response from the LLM to automatically call already defined functions. If you want the LLM to know the user's name, you can define an additional function to provide the user's information, but in my experience, it doesn't tend to actively call functions. Of course, you can force a function call, but this will not generate any other prompts other than the function call, and if you want to introduce function calls into your regular prompting, you'll have to find another way to do it, since you won't know what form that function prompt will take, which will further reduce your token limit.

For now, LLMs that force function calls are defined (summarization, vectorstore query, browsing), and LLMs that respond directly to the user are designed to respond based on the results generated in the previous step.