Language Support for Agent Names in GroupChat

[erthman18]

When building a session using GroupChat, the agent name does not support using Chinese, and an error is reported: openai. error. InvalidRequestError: 'User' does not match '^ [a-zA-Z0-9_ -] {1,64} $' - 'messages. 1. name' Directly using initiate does not have this problem

[LittleLittleCloud]

The error is from openai side, where the legitimate name has to match ^ [a-zA-Z0-9_ -] {1,64}

from the openai cookbook, it seems that the name field is for function name only, so we need to avoid setting name field when message's role is not a function

To inject agent name information into message in group chat, we can add the name information as suffix separated by a special token

howdy
<eof_msg>
From xxx // xxx is agent name

and set stop word to <eof_msg> to prevent the suffix being generated in llm call.

[sonichi]

@LittleLittleCloud I don't see any mention that "name is for function name only". Where do you see that?

[LittleLittleCloud]

The only example from that cookbook that contains name field is the function call.

Beside, from the below document, the name is required only when role is function. It did mention that the name can be the author of message. But based on my experiment, the name field is actually not respected even when you provide it.

The experiment is simply sending the following message to openai

{
   role: system
   content: You are a helpful assistant
   name: Alice
}
{
   role: user
   content: what's my name
   name: Bob
}

Received:

I'm sorry, but as an AI assistant, I don't have access to personal information unless you provide it to me.

[sonichi]

Can anyone run an experiment to compare:

1. the current group chat implementation
2. @LittleLittleCloud 's suggestion about adding name to the end of each msg

with all the three group chat notebook examples, and the 12 examples in the paper?

[yiranwu0]

What I understand is that name can be used for messages other than functions, and the names will be passed in the model (the token will be counted)

There are some observations here: https://community.openai.com/t/discovered-how-a-name-is-added-to-api-chat-role-messages-and-the-tokens/330016

[LittleLittleCloud]

Thanks @kevin666aa It's a really helpful link to share here.

So I does some extra experiments on dotnet, typescript and curl. My observation is it's unclear how GPT processes the name field. We can create another thread to explore the name field though. As it would be another topic

var systemMessage = new ChatMessage(
    role: ChatRole.System,
    content: "You are a helpful AI assistant")
{
    Name = "Alice",
};

var userMessage = new ChatMessage(ChatRole.User, "Hey what's your name")
{
    Name = "Bob",
};

var option = new ChatCompletionsOptions()
{
    Temperature = 0,
};

option.Messages.Add(systemMessage);
option.Messages.Add(userMessage);

var response = await Constant.AzureOpenAI.GetChatCompletionsAsync(Constant.GPT_35_MODEL_ID, option);

// response
// Hello Bob! I am Alice, your helpful AI assistant. How can I assist you today?

var msgs = [{
    role: 'system',
    content: 'hello',
    name: 'alice',
},
{
    role: 'user',
    content: 'what is my name',
    name: 'bob',
}];
var choices = await client.getChatCompletions(
    AZURE_GPT_3_5_TURBO_16K!,
    msgs,
    {
        temperature: 0
    }
);

// I'm sorry, but I don't know your name.

curl https://api.openai.com/v1/chat/completions `
  -H "Content-Type: application/json" `
  -H "Authorization: Bearer your token" `
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant.",
        "name": "Alice"
      },
      {
        "role": "user",
        "content": "what is my name",
        "name": "Bob"
      }
    ]
  }'

// response
I'm sorry but I don't have access to personal information about individuals unless it has been shared with me in the course of our conversation. I am designed to respect user privacy and confidentiality. My primary function is to provide information and answer questions to the best of my knowledge and abilities. If you have any concerns about privacy or data security, please let me know, and I will do my best to address them.

So we can see that in typescript, dotnet and curl, the behavior is different. Would be appreciated if anyone can verify that code as well.

In autogen, here's what I get. We can see the agent name is not recognized.

Also trying some magic prompts in case the name is unknown is because of privacy policy. No lucky here

Again, would be appreciated if anyone can re-run the code snippet as well.

[cpacker]

Hi @sonichi quick question (apologies if this is easily answered via browsing the source code) - in AutoGen do you assume that the OpenAI compatible backend is using the name field in a smart way (eg by prefixing in the prompt etc)?

Or does AutoGen do some sort of prompt formatting using the name field before sending things over to the OpenAI compatible backend?

From my reading of POST requests being sent by AutoGen while testing AutoGen+MemGPT (+ reading this issue), I'm assuming it's the former?

I'm asking because I recently added some of my own prompt formatting support for this in the MemGPT local LLM prompt formatter: https://github.com/cpacker/MemGPT/pull/603 (eg \nUSER: -> \nUSER (autogen_name): for the Alpaca style formatter).

[sonichi]

Yes, it's the former. Your approach looks smart.

[thinkall]

We are closing this issue due to inactivity; please reopen if the problem persists.