freedeaths
commented
7 months ago What I implemented now are as follows:
- Define the finite_state_machine like:
- Now the on condition is not used.
{ "agents": [agent1, agent2, agent3], "transitions": [ {"from": agent1, "to": agent2, "on": "event1"}, {"from": agent2, "to": agent3, "on": "event2"} ] }
- Now the on condition is not used.
- Implement a new method fsm_select_speaker in GroupChat
- If there is only one available agent, return it as selected_agent to avoid LLM calling.
- Otherwise, let LLM pick up a reasonable agent.
def fsm_select_speaker(self, last_speaker: Agent, agents: Optional[List[Agent]] = None) -> Union[Agent, None]:
"""Select the next speaker by pre-defined finite state machine."""
if agents is None:
agents = self.agents
to_agents = [t["to"] for t in self.finite_state_machine['transitions'] if t['from'] == last_speaker]
if len(to_agents) == 1:
return to_agents[0], to_agents
elif len(to_agents) == 0:
logger.warning(
f"There is no successor from {last_speaker}."
"It is not expected and all the agents will be returned."
)
return None, agents
else:
# TODO: The case from last_speaker to last_speaker has not been handled yet
if self.allow_repeat_speaker == True:
to_agents.append(last_speaker)
return None, to_agents- If self.speaker_selection_method.lower() == "fsm" and self.finite_state_machine == None raise an Error
- Implement the logic as follows. In a finite state machine, state transitions are only related to the previous state and are independent of any earlier states. Therefore, it is only necessary to consider the last message from the messages. There is no need to add an additional system prompt, which can effectively increase the likelihood of LLM making the correct selection.
#select_speaker_messages = None
#if self.speaker_selection_method.lower() == "manual":
# selected_agent = self.manual_select_speaker(agents)
#elif self.speaker_selection_method.lower() == "round_robin":
# selected_agent = self.next_agent(last_speaker, agents)
#elif self.speaker_selection_method.lower() == "random":
# selected_agent = random.choice(agents)
elif self.speaker_selection_method.lower() == "fsm":
selected_agent, agents = self.fsm_select_speaker(last_speaker, agents)
select_speaker_messages = self.messages.copy()
if select_speaker_messages[-1].get("function_call", False):
select_speaker_messages[-1] = dict(select_speaker_messages[-1], function_call=None)
if select_speaker_messages[-1].get("tool_calls", False):
select_speaker_messages[-1] = dict(select_speaker_messages[-1], tool_calls=None)
select_speaker_messages = select_speaker_messages[-1:]
#else:
# ...In this implementation and experiment result, the FSM is quite successful. Moreover, it is a relatively universal organizational method for Groupchat, and it does not cause much breaking to the previous code.
So may I submit a PR to contribute the implementation method of this FSM?
victordibia
afourney
IANTHEREAL
Is your feature request related to a problem? Please describe.
Last week, in an effort to utilize Augoten’s Groupchat for addressing production scenarios, I embarked on a series of experiments to evaluate the precision and dependability of the outcomes. Despite numerous tweaks to the prompts, encompassing the system_prompt and descriptions for each role, particularly the descriptions, I observed that the manager was still struggling to select the next speaker effectively as per the guidelines in the description.
To scrutinize the manager’s decision-making capability in choosing the next speaker in isolation, I devised a more streamlined experiment.
The system messages from each agent outlined above delineate the task of sequential counting. To focus on assessing the manager’s capabilities, I’ve designed the task to be as straightforward as possible. Moreover, I’ve conducted tests using GPT-3.5 to verify whether a number under 30 is a multiple of 3 or 5, and found that the LLM performance is up to par.
The descriptions provided by each agent illustrate the state transitions as shown in the following diagram, essentially aiming to evaluate if the manager can select the next speaker in accordance with the finite state machine’s stipulations.
I ran the code locally using GPT-4-0613 and GPT-4-1106-preview 2 times and 4 times respectively. The results of all 6 runs were not as expected.
The output of GPT-4-0613 is:
The output of GPT-4-1106-preview is:
And then I found PR 857. So I implemented the following code:
And I still get the wrong results from both GPT-4 and GPT-4-1106-preview.
The output of GPT-4-0613 is:
The output of GPT-4-1106-preview is:
Describe the solution you'd like
Therefore, I want to implement a new speaker_selection_method method, called fsm, which stands for "finite state machine".
The usage is to define a finite state machine fsm after initializing each Agent, and before initializing GroupChat, and then set speaker_selection_method="fsm" when initializing GroupChat, and introduce a new parameter finite_state_machine (finite_state_machine defaults to None, so it does not break the original design and usage)
The usage is like:
Additional context
I have implemented the method locally, and then repeated the experiment with gpt-4-1106-preview, with a success rate of 10/10 = 100%.