Inject ChatMemory into the Prompt

[andreadimaio]

Discussed in https://github.com/quarkiverse/quarkus-langchain4j/discussions/879

^{Originally posted by **andreadimaio** September 9, 2024} Hi, I'd like to share an idea that could speed-up the process of creating a prompt that leverage `ChatMemory` for AI interactions. The focus of this proposal is to simplify how developers pass conversation history into prompts. ## Problem Currently, when developing an AI service that utilizes a conversation history as part of the prompt, the developer is responsible for manually formatting the chat history. This involves extracting and formatting past `user` and `assistant` messages into a structured format. Consider the example below, where the developer needs to inject a conversation history (as a formatted string) into the prompt: ```java @RegisterAiService( chatMemoryProviderSupplier = NoChatMemoryProviderSupplier.class ) public interface AIService { @SystemMessage(""" Given the following conversation and a follow-up question, rephrase the follow-up question to be a standalone question, in its original language. Return the follow-up question VERBATIM if the question is changing the topic with respect to the conversation. It is **VERY IMPORTANT** to only output the rephrased standalone question; do not add notes or comments in the output. Chat History: {chatHistory}""") public String rephrase(String chatHistory, @UserMessage String question); } ``` Where the `chatHistory` placeholder has the format: ``` User: Assistant: ... ``` To implement this example, the developer needs to inject the `ChatMemoryStore` in a service class and generate the correct format for the `chatHistory` placeholder. ## Simplifying ChatMemory Injection To accelerate this process and reduce manual work, it could be useful to have a new feature that allows the `ChatMemoryStore/ChatMemory` to be passed directly as a method parameter. In this case a specific implementation will be used to automatically map the `ChatMemoryStore/ChatMemory` to the appropriate format that the developer requires for the prompt, where the default format could be what I wrote above. ```java @RegisterAiService( chatMemoryProviderSupplier = NoChatMemoryProviderSupplier.class ) public interface AIService { @SystemMessage(""" Given the following conversation and a follow-up question, rephrase the follow-up question to be a standalone question, in its original language. Return the follow-up question VERBATIM if the question is changing the topic with respect to the conversation. It is **VERY IMPORTANT** to only output the rephrased standalone question; do not add notes or comments in the output. Chat History: {chatHistory}""") public String rephrase(ChatMemoryStore chatHistory, @UserMessage String question); } ``` ### Customizable Format: If the developer requires a different conversation format, they could customize the formatting by adding a property in the `@RegisterAiService` annotation. For example: ```java @RegisterAiService( chatMemoryProviderSupplier = NoChatMemoryProviderSupplier.class , chatFormat = CustomChatFormat.class ) public interface AIService { ... } ``` This is an idea that came to mind while using quarkus-langchain4j. I don't know if it makes sense or if there is already something that can be used to do this operation, let me know 😄.

[geoand]

cc @cescoffier

[cescoffier]

If instead of injecting the ChatMemoryStore, we inject a list of ChatMessage, can't we use plain Qute to format the prompt?

{#for m in chatHistory}
   {#if m.type.name() == "USER"} 
   User: {m.text()}
   {/if}
   {#if m.type.name() == "AI"} 
   Assistant: {m.text()}
   {/if}
{/for}

I vonluntary filtered out system and tools.

[andreadimaio]

Being able to use something like Jinja is very interesting, but from my point of view, I prefer to keep the prompt simple and just use a placeholder.

[cescoffier]

We could imagine a fragment or an include to achieve this. HEre is an example using an "include":

 @SystemMessage("""
        Given the following conversation and a follow-up question, rephrase the follow-up question to be a
        standalone question, in its original language. Return the follow-up question VERBATIM if the
        question is changing the topic with respect to the conversation. It is **VERY IMPORTANT** to only
        output the rephrased standalone question; do not add notes or comments in the output.

        Chat History:
        {#include history limit=10}""")
    public String rephrase(ChatMemoryStore chatHistory,  @UserMessage String question);

The "included" template named "history" would have access to the chatHistory object. In my example, I passed an extra optional parameter, if, for example, we want to limit the number of messages.

The history template would be a separated file:

{#for m in chatHistory}
   {#if m.type.name() == "USER"} 
   User: {m.text()}
   {/if}
   {#if m.type.name() == "AI"} 
   Assistant: {m.text()}
   {/if}
{/for}

(I didn't include the limit support, but you see the idea).

I'm not against using a toString on the chat memory object, but we need to be very careful with this approach as it can end badly (object reference, cycles, etc.). The template approach is similar to what quarks langchain4j is already doing.

[cescoffier]

BTW, the quarkus-langchain4j prompts are already templates. So, what I proposed should work already.

It uses Qute.

[andreadimaio]

I tried this prompt and it works:

@SystemMessage("""
        Given the following conversation and a follow-up question, rephrase the follow-up question to be a
        standalone question, in its original language. Return the follow-up question VERBATIM if the
        question is changing the topic with respect to the conversation. It is **VERY IMPORTANT** to only
        output the rephrased standalone question; do not add notes or comments in the output.

        Chat History:
        {#for m in chatHistory}
            {#if m.type.name() == "USER"}
                User: {m.text()}
            {/if}
                {#if m.type.name() == "AI"}
            Assistant: {m.text()} 
            {/if}
        {/for}""")
public String rephrase(List<ChatMessage> chatHistory, @UserMessage String question);

This is not exactly what I wrote in my proposal, but in the end it works just fine (without adding any more complexity).

[geoand]

We could even make chatHistory available by default to the template.

WDYT?

[andreadimaio]

Something like {response_schema}? Cool!

[geoand]

Something like {response_schema}

Sorry, I am not following this :). Care to elaborate?

[andreadimaio]

{response_schema} is a placeholder that you can use directly in the template. If I understand you correctly, you want to give the developers a new placeholder like chat_history, which will automatically be converted to the user/assistant format. If the developer wants to change the format they can use Qute. Is that right? :)

[geoand]

Oh, I didn't remember we had that placeholder!

Yeah, that's exactly the intent of what I proposed.

[andreadimaio]

I like it! :)

[cescoffier]

@geoand you learn very quickly about {response_schema} as soon you do not use OpenAI:-)

[geoand]

😄

[geoand]

I have opened https://github.com/quarkiverse/quarkus-langchain4j/pull/887. Would you like to add a commit that adds a test and the docs you have in mind for your use case?

Thanks