[Issue]: TransformMessages not working with GroupChatManager speaker selection prompts

[Nathan-Intergral]

Describe the issue

When GroupChat speaker_selection_method is set to 'auto' the speaker selection prompt used to determine the next agent contains all the messages in the conversation history. Attempting to add TransformMessages to the GroupChatManager should allow you to limit the length of each speaker selection prompt but it does not.

This is due to the select_speaker function in groupchat.py uses conversable_agent.py's generate_oai_reply. Which unlike generate_reply does not call process_all_messages_before_reply so the hook for transform messages is never run.

Steps to reproduce

1. Setup group chat and manager:

group_chat = GroupChat(
            agents=agents,
            messages=messages,
            max_round=max_round,
            speaker_selection_method="auto",
            send_introductions=introductions,
            speaker_transitions_type="allowed",
            allowed_or_disallowed_speaker_transitions=get_allowed_transitions(workers),
        )

        manager = GroupChatManager(
            groupchat=group_chat,
            llm_config=get_llm_config(),
            is_termination_msg=lambda x: "GROUPCHAT_TERMINATE" in x.get("content", ""),
        )

        transform_messages.TransformMessages(
            transforms=[
                transforms.MessageHistoryLimiter(max_messages=5),
            ]
        )

2. initialise groupchat conversation with runtime logging enabled

logging_session_id = autogen.runtime_logging.start(config={"dbname": "logs.db"})

await overseer.a_initiate_chat(
                        manager,
                        clear_history=False,
                        message=new_request,
                    )

autogen.runtime_logging.stop()

3. look at chat completions in logs.db, look for speaker selection prompts and note that message history provided in prompt is not limited to 5 and will continuously grow in token count as the conversation continues. Please note that this does not work for message limits or token limits.

Screenshots and logs

No response

Additional Information

No response

[Nathan-Intergral]

@WaelKarkoub @ekzhu Created as discussed

[ekzhu]

Let's start this after #2304 is merged

[ekzhu]

@Nathan-Intergral we are discussing a solution in this issue https://github.com/microsoft/autogen/issues/2583. Once the solution is merged we can close this issue.

[WebsheetPlugin]

I am sorry, but TransformMessages is not working in group chat. Was it not working all the time or is there a version it works??

[WaelKarkoub]

@WebsheetPlugin we are working on a solution. cc @marklysze

[WebsheetPlugin]

@WaelKarkoub I got it. Thanks a lot.

I am not sure if it is related but I created this report: https://github.com/microsoft/autogen/issues/2638#issue-2288327906

For me transform works in group chat, but the transform results are not chained.

[marklysze]

I'll have a look at this and start working on it... (adding the TransformMessages ability to the select speaker)

[marklysze]

Just an update that we have started this and I'm working with @WaelKarkoub on some tweaks to the TransformMessages around filtering messages so that we can allow the user to target specific messages for transforms, which is key for the select speaker messages.

[marklysze]

Hey @Nathan-Intergral, just a note that I've created a branch under AutoGen which will be for the addition of the speaker selection transforms functionality: https://github.com/microsoft/autogen/tree/selectspeakertransforms

At this stage it's functional, in that you can add transforms to the group chat by setting the select_speaker_auto_message_transforms property of a groupchat to a TransformMessages object.

E.g.

debate_select_speaker_compression_args = dict(model_name="microsoft/llmlingua-2-xlm-roberta-large-meetingbank", use_llmlingua2=True, device_map="cpu")

debate_select_speaker_transforms = transform_messages.TransformMessages(
    transforms=[
        transforms.MessageHistoryLimiter(max_messages=10),
        transforms.MessageTokenLimiter(max_tokens=3000, max_tokens_per_message=500, min_tokens=300),
        transforms.TextMessageCompressor(
            min_tokens=1000,
            text_compressor=transforms.LLMLingua(
                debate_select_speaker_compression_args, structured_compression=True
            ),
            cache=InMemoryCache(seed=43),
            filter_dict={"role": ["system"], "name": ["Debate_Moderator_Agent", "checking_agent"]},
            exclude_filter=True,
        ),  # Allows use of <llmlingua, compress=False></llmlingua> to stop compressing certain content
    ]
)

group_chat = autogen.GroupChat(
    agents=self.agent_list,
    messages=[],
    max_round=8,
    select_speaker_message_template="You are managing a debate and your only job is to select the next speaker, each speaker has a name. Follow the debate and decide on who should speak next. The 'Debate_Moderator_Agent' is the first speaker and they will kick off the debate with a topic to debate. Then each of the four debaters will speak and speak only once each. You should start by selecting the 'Affirmative_Constructive_Debater' to provide their opening arguments in the debate.",
    select_speaker_prompt_template="Read the above conversation and your job role, which is managing the debate and choosing the next speaker. The valid speakers can be selected from this list {agentlist}. During the debate the order of debaters are: 1st is the 'Affirmative_Constructive_Debater', 2nd is the 'Negative_Constructive_Debater', 3rd is the 'Affirmative_Rebuttal_Debater', and 4th is the 'Negative_Rebuttal_Debater'. Then 5th will be the 'Debate_Judge' and 6th is the 'Debate_Moderator_Agent'.",
    max_retries_for_selecting_speaker=1,
    role_for_select_speaker_messages="user", # Ensure select speaker messages are role='user'
    select_speaker_auto_verbose=True,
    select_speaker_auto_message_transforms=debate_select_speaker_transforms,
)

I'll create a PR based on this branch, but please note that some of the filtering functionality (shown in the sample code above) is dependent on another PR being merged, #2661.

[marklysze]

Okie dokie, #2719 has been created!

[Nathan-Intergral]

@marklysze Great thank you for that!