Allow the AiService to inject AiMessage into the prompt

[andreadimaio]

Might be useful to have an annotation that allows the developer to use a template for the AiMessage.

Example:

@RegisterAiService
public interface AiService {

    @SystemMessage("You are a helpful assistant")
    @UserMessage("Input: {question}")
    @AiMessage("Output: ")
    public String answer(String question);
}

This is particularly useful if you are using models that use tags. I don't know if this is something that might come from langchain4j.

[andreadimaio]

For example if I'm using llama-3-1-70b-instruct the output should be:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>
You are a helpful assistant<|eot_id|><|start_header_id|>user<|end_header_id|>
Input: Hi<|eot_id|><|start_header_id|>assistant<|end_header_id|>
Output:

[geoand]

Hm, again I wonder if this is something that would be handled by @SeedMemory

[andreadimaio]

No, at least I don't think so from what I know about @SeedMemory.

The @SeedMemory helps me to inject some data into the ChatMemory in the format I prefer, so this help me to have an AiMessage with the correct prefix, but what happens when a new message is sent to the LLM? The "Output: " will disappear.

[geoand]

Fair enough.

Question: wouldn't having AiMessage after @UserMessage and without another user input break the inference? I think I recall OpenAI requiring a specific order of messages.

[andreadimaio]

I see your point... in this case the AiMessage specified in the AiService is something that is only used to get the input message that is sent to the LLM, it does not have to be part of the memory.

[andreadimaio]

What I'm tryng to say is, in the example above the result of the memory after a call should be something like:

SystemMessage: You are a helpful assistant
UserMessage: Input: {question}
AiMessage: Output: <value_returned_by_llm> 

and not:

SystemMessage: You are a helpful assistant
UserMessage: Input: {question}
AiMessage: Output:
AiMessage: <value_returned_by_llm> 

[geoand]

I see. @langchain4j is this something that would make sense for the project?

[andreadimaio]

I'm thinking more about the scenarios where I could use the @AiMessage annotation in the AiService and they are all "one shot calls" (LLM without memory). In this case I can get a "good" prompt without the use of the @AiMessage.

I understand that for providers using a "chat" API this can be a bit weird. In any case, I know how to go on without adding this feature, from my side it is also possible to close this issue. :)

[geoand]

Thanks for the update.

Let's see what @langchain4j thinks

[langchain4j]

Hi @andreadimaio, could you provide a (real world) example when this can be useful?

There is a related request for Anthropic, but there we agreed to handle it on ChatLanguageModel level. It seems that it might indeed be useful to have something like @AiMessage, or more precisely, @AiMessagePrefix annotation.

[andreadimaio]

We can use the first message as example.

Let's say I'm using llama-3-1-70b-instruct and I need to construct a prompt with a @SystemMessage, @UserMessage and to get the best response from the model it's useful to also send a prefix for the @AiMessage in the request body.

So suppose the request to get a best answer has to be formatted like this:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>
You are a helpful assistant<|eot_id|><|start_header_id|>user<|end_header_id|>
Input: Hi<|eot_id|><|start_header_id|>assistant<|end_header_id|>
Output:

Where:

• You are a helpful assistant is part of the @SystemMessage,
• Input: Hi is part of the @UserMessage
• Output: is part of the @AiMessage

Today, it is not possible to use the annotations in the AiService to achieve this behaviour, because if I write something like this

@RegisterAiService
public interface AiService {

    @SystemMessage("You are a helpful assistant")
    @UserMessage("Input: {question}\nOutput:")
    public String answer(String question);
}

the result will be:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>
You are a helpful assistant<|eot_id|><|start_header_id|>user<|end_header_id|>
Input: Hi\nOutput:<|eot_id|><|start_header_id|>assistant<|end_header_id|>

As you can see in this case, the "Output:" prefix is in the @UserMessage and it is not what I'm looking for.

[andreadimaio]

From my point of view, the @AiMessagePrefix is a good solution to solve this scenario.

[langchain4j]

@andreadimaio sorry for confusion, but I am still not sure what is the purpose of "Output:" here? Is it to nudge the model to start the answer with this specific text, or is it to follow the formatting of how the model was fine-tuned (can't find "Output:" in the documentation)?

[andreadimaio]

Is it to nudge the model to start the answer with this specific text, or is it to follow the formatting of how the model was fine-tuned (can't find "Output:" in the documentation)?

This is just an example, but yes the ability to add a prefix gives the developer more flexibility when writing a prompt, and as you said, some models could be fine-tuned to respond better with a prefix.

[langchain4j]

@andreadimaio sure, let's go with @AiMessagePrefix. But I am still curious about real-world example of a prefix, cause "Output:" does not seem particulary useful.

[andreadimaio]

@andreadimaio sure, let's go with @AiMessagePrefix. But I am still curious about real-world example of a prefix, cause "Output:" does not seem particulary useful.

No, personally I don't have any real-world scenario, as I wrote in some message above, my use cases can do without the @AiMessagePrefix. However, I think this is a good method to customise the prompt to send to the LLM :)