sonichi
commented
9 months ago @afourney FYI
Closed shuther closed 5 months ago
sonichi
commented
9 months ago @afourney FYI
afourney
commented
9 months ago Interesting. I'll add it to the speaker selection tests and see if I can replicate it. It's unclear at which stage this is happening.
One of my major test cases (human eval with a third agent) uses underscores in the agent name without any issue. So I am skeptical it comes from GroupChat itself. Maybe the model is doing it?
shuther
commented
9 months ago So it seems there are several issues. I am trying to replicate the notebook agentchat_groupchat_RAG.ipynb into a normal python program with other models.
'\nRole: Senior\\_Python\\_Engineer\n\n---\n\nYou are a senior python engineer, and you have been tasked with providing guidance on how to use Spark for parallel training in FLAML. You will be providing sample code to illustrate your points.\n\nTo use Spark for parallel training in FLAML, you can leverage the PySpark MLlib library, which provides a high-level API for building and training machine learning models using Spark. Here is an example of how you might use PySpark MLlib to train a simple linear regression model using Spark\'s distributed computing capabilities:\n```python\nfrom pyspark.ml import LinearRegression\nfrom pyspark.sql import SparkSession\n\n# Create a Spark session\nspark = SparkSession.builder.appName("Parallel Training Example").getOrCreate()\n\n# Load the data into a DataFrame\ndata = spark.read.format("csv").option("header", "true").load("data.csv")\n\n# Split the data into training and testing sets\ntrain_data, test_data = data.randomSplit([0.7, 0.3])\n\n# Create a Linear Regression model using PySpark MLlib\nmodel = LinearRegression().setMaxIter(10).setRegParam(0.3)\n\n# Train the model on the training data\nmodel.fit(train_data)\n\n# Evaluate the model on the testing data\npredictions = model.transform(test_data)\nrmse = predictions.stat.rootMeanSquaredError()\nprint("Root Mean Squared Error: ", rmse)\n```\nIn this example, we first create a Spark session and load the data into a DataFrame. We then split the data into training and testing sets using therandomSplit()method. Next, we create a Linear Regression model using PySpark MLlib and set its maximum number of iterations and regularization parameter. Finally, we train the model on the training data using thefit()method and evaluate its performance on the testing data using thetransform()method and thestat.rootMeanSquaredError()method.\n\nNote that this is just a simple example to illustrate how to use PySpark MLlib for parallel training in Spark. In practice, you would likely need to modify this code to suit your specific use case and data.'--------------------------------------------------------------------------------
WARNING:autogen.agentchat.groupchat:GroupChat select_speaker failed to resolve the next speaker's name. This is because the speaker selection OAI call returned:
Product\_Manager
Product_Manager (to chat_manager):
TERMINATE
--------------------------------------------------------------------------------
WARNING:autogen.agentchat.groupchat:GroupChat select_speaker failed to resolve the next speaker's name. This is because the speaker selection OAI call returned:
Code\_Reviewer
phoenixbai
commented
9 months ago any update on this? I try running agentchat_groupchat.ipynb, and encounters the same issue:
GroupChat select_speaker failed to resolve the next speaker's name. This is because the speaker selection OAI call returned:
AndreiD
commented
9 months ago same
Maxime-Charmasson
commented
9 months ago Pretty much the same thing happens when I want to use Groupchat. In the beginning, something like the following is sometimes generated "â[33mContentCreatorâ[0m (to chat_manager):". A friend saw the following on his screen : "\033[34m" and "\033[0m". These look like colour codes? And then it goes all wrong, because the Groupchat manager cannot find the next speaker, and then further generates "GroupChat select_speaker failed to resolve the next speaker's name. This is because the speaker selection OAI call returned:" without a speaker, while there is still some output from a speaker, but from which speaker is not clear. Until this bug (and related issues) gets fixed, Autogen is pretty much worthless.
lightfinesthour
commented
8 months ago same issue, cleaning up the cache at every run does not resolve the issue.
AssetDK
commented
8 months ago same issue I am not sure what to report here to help solve the issue. I am running four agents, and using LM studio as the server. It looks like the agents chat either break off, requestion a prompt or it loops around getting similar responses. I don't know if this helps - say this is the model:
{ "name": "mistralai_mistral-7b-instruct-v0.2", "arch": "llama", "quant": "Q4_K_S", "context_length": 32768, "embedding_length": 4096, "num_layers": 32, "rope": { "freq_base": 1000000, "dimension_count": 128 }, "head_count": 32, "head_count_kv": 8, "parameters": "7B" }
and this is a snip of the chat: 2023-12-23 00:08:47.054] [INFO] [LM STUDIO SERVER] Context Overflow Policy is: Stop at limit [2023-12-23 00:08:47.054] [INFO] Provided inference configuration: { "n_threads": 4, "n_predict": -1, "top_k": 40, "top_p": 0.95, "temp": 0.8, "repeat_penalty": 1.1, "input_prefix": "[INST]", "input_suffix": "[/INST]", "antiprompt": [ "[INST]" ], "pre_prompt": "You are in a role play game. The following roles are available:\nDirector: You are the director of our team. The goal of the team is to build a project specific prompt ..\n\nRead the following conversation.\nThen select the next role from ['Director', 'Assistant', 'Critic', 'AgentBuilder'] to play. Only return the role.", "pre_prompt_suffix": "", "pre_prompt_prefix": "", "seed": 1111, "tfs_z": 1, "typical_p": 1, "repeat_last_n": 64, "frequency_penalty": 0, "presence_penalty": 0, "n_keep": 0, "logit_bias": {}, "mirostat": 0, "mirostat_tau": 5, "mirostat_eta": 0.1, "memory_f16": true, "multiline_input": false, "penalize_nl": true }
It looks like the UserproxyAgent gets a somewhat "confusing" pre_promp: "You are in a role play game. The following roles are available:\nDirector:" .... and then the Director is prompted to select: "Then select the next role from ['Director', 'Assistant', 'Critic', 'AgentBuilder'] to play. Only return the role."
I wonder if a change to the prepompt texst would help? [](https://discord.com/channels/1110598183144399058/1177683199514660934/1187733442977337365 Link to LM Studio Discord regarding the code used)
Maxime-Charmasson
commented
8 months ago @AssetDK Link to discord channel is not correct I think (resulting in no text channels).
mongolu
commented
8 months ago I'm hitting this problem also. What can be done to resolve it?
afourney
commented
8 months ago Hi everyone. I'm looking into this this now. Notably, I would expect GroupChat to work poorly on everything but GPT-4. It looks like at least on example is using Mistral. I'm not sure what models everyone else is using -- please include this information.
@AssetDK as for the confusing "pre-prompt" -- I 100% agree. I tried to have that fixed a while back, but made the mistake of including it in another PR, then removing it so that review would go more smoothly: https://github.com/microsoft/autogen/pull/736#issuecomment-1844797374 I'll follow-through and submit the prompt fix as a standalone PR.
I am also a little confused by this pre-prompt?
"pre_prompt": "You are in a role play game. The following roles are available:\nDirector: You are the director of our team. The goal of the team is to build a project specific prompt ..\n\nRead the following conversation.\nThen select the next role from ['Director', 'Assistant', 'Critic', 'AgentBuilder'] to play. Only return the role.",
There should be at least 4 agents listed, not just the Director. Can you provide an example of how you initialize your agents and your GroupChat? In particular, are any of your agents using function calling?
AssetDK
commented
8 months ago Hi everyone. I'm looking into this this now. Notably, I would expect GroupChat to work poorly on everything but GPT-4. It looks like at least on example is using Mistral. I'm not sure what models everyone else is using -- please include this information.
Really appreciate this! Im running LM studio version with TheBloke/Mistral-7B-Instruct-v0.2-GGUF/mistral-7b-instruct-v0.2.Q4_K_S.gguf Also tried: TheBloke/stablelm-zephyr-3b-GGUF/stablelm-zephyr-3b.Q6_K.gguf - similar result!
Below you find is my entire code - I tried to add an extra agent "Builder" a meta-agent that I hope can suggest new agent definitions. These tree/four agents should illustrate how I could use the agents to explore what the LLM knows about some EU legislation. I have some ideas for MUCH more interesting teams, but that will have to wait... :-)
I wonder, as I am not toooo familiar with Python syntax, if the "extra" commas in the parameter lists could mess things up ['Director', 'Assistant', 'Critic', 'AgentBuilder'] but it did Little difference with or without! (Also I wonder how the code is reset, it looks like some answers are preserved between sessions. )
(from autogen import AssistantAgent, UserProxyAgent, GroupChat, GroupChatManager
config = {
"config_list": [
{
# "base_url": "http://localhost:5001/v1",
"base_url": "http://localhost:1234/v1",
"api_key": "NULL"
}
],
"seed": 1111
}
director = UserProxyAgent(
"Director",
code_execution_config=False,
human_input_mode="NEVER",
system_message="You are the director of our team. The goal of the team is actually to describe the roles of a new autogen Agent expert team! This team will be working with the implementation if EU's CAP legislation, locally in denmark. Based on the Critic and your skills you will deside when the task is done and when to request more input." ## "When your ready, or once in a while, you must ask AgentBuilder to make the code for new autogen AssistantAgent descriptions for our new autogen Agent expert team!",
default_auto_reply="Very fine lets revisit our plans!", max_consecutive_auto_reply=5
)
assistant = AssistantAgent(
"Assistant",
system_message="Assistant to the Director. You will build a full orderede list of possibel experts and a tasklist for the skills and tools these experts may need, that is our new autogen Agent expert team regarding EU's CAP legislation.",
llm_config=config,
)
critic = AssistantAgent(
name="Critic",
system_message="You will inspect the liste of experts plans made for gatering expert knowlege regarding EU's CAP legislation. You will point out gaps and even suggest new taks to improve quality!",
llm_config=config,
)
builder = AssistantAgent(
name="AgentBuilder",
system_message="You are an expert in openai and autogen. Your task is to build new autogen AssistantAgent descriptions for the agents that will be needed for our project.",
llm_config=config,
)
groupchat = GroupChat(
agents=[
director,
assistant,
critic,
##builder,
],
messages=[],
max_round=15,
speaker_selection_method="auto",
allow_repeat_speaker=True,
##speaker_selection_method="manual"
)
manager = GroupChatManager(
name="Chat",
groupchat=groupchat,
llm_config=config
)
director.initiate_chat(
manager,
message="Please workout a plan for AgentBuilder so he can can describe a new team of autogen AssistantAgent agents that are experts on EU's CAP legislation. Starting from what we know about EU's CAP, what are the tasks and skills we need and how can we best describe such a team of autogen AssistantAgent agents?"
))@AssetDK Link to discord channel is not correct I think (resulting in no text channels). Perhaps this works better - will edit the original message! - I also guess you need to subscribe to LM studio discord to go there directly! https://discord.com/channels/1110598183144399058/1177683199514660934/1187733442977337365
Maxime-Charmasson
commented
8 months ago Hi Per Feldvoss Olsen @AssetDK https://github.com/AssetDK
The link is still resulting in âNO TEXT CHANNELSâ.
Van: Per Feldvoss Olsen @.> Verzonden: jeudi 28 décembre 2023 15:04 Aan: microsoft/autogen @.> CC: Maxime-Charmasson @.>; Comment @.> Onderwerp: Re: [microsoft/autogen] Possible Regression - select_speaker failed to resolve the next speaker's name - (Issue #842)
@AssetDK https://github.com/AssetDK Link to discord channel is not correct I think (resulting in no text channels). Perhaps this works better - will edit the original message! https://discord.com/channels/1110598183144399058/1177683199514660934/1187733442977337365
â Reply to this email directly, view it on GitHub https://github.com/microsoft/autogen/issues/842#issuecomment-1871205721 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AVBNRQTZ3TOHM52FGOXQNCLYLV34DAVCNFSM6AAAAABAEBLW5KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZRGIYDKNZSGE . You are receiving this because you commented. https://github.com/notifications/beacon/AVBNRQTWSX7VEDVQANOCCR3YLV34DA5CNFSM6AAAAABAEBLW5KWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTTPRBKVS.gif Message ID: @. @.> >
AssetDK
commented
8 months ago @afourney - When I disregard the original problem it looks like things are working "somewhat"... until I get this Trackback message, I guess that the code could be fragile if it depends on the reply syntax and try to convert in to messages.
Traceback (most recent call last):
File "d:\MYPhyton\Autogen1\autogentest\autogen_EU_CAP_1.py", line 61, in <module>
director.initiate_chat(
File "C:\Users\asset\anaconda3\Lib\site-packages\autogen\agentchat\conversable_agent.py", line 550, in initiate_chat
self.send(self.generate_init_message(**context), recipient, silent=silent)
File "C:\Users\asset\anaconda3\Lib\site-packages\autogen\agentchat\conversable_agent.py", line 348, in send
recipient.receive(message, self, request_reply, silent)
File "C:\Users\asset\anaconda3\Lib\site-packages\autogen\agentchat\conversable_agent.py", line 481, in receive
reply = self.generate_reply(messages=self.chat_messages[sender], sender=sender)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\asset\anaconda3\Lib\site-packages\autogen\agentchat\conversable_agent.py", line 906, in generate_reply
final, reply = reply_func(self, messages=messages, sender=sender, config=reply_func_tuple["config"])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\asset\anaconda3\Lib\site-packages\autogen\agentchat\groupchat.py", line 269, in run_chat
speaker = groupchat.select_speaker(speaker, self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\asset\anaconda3\Lib\site-packages\autogen\agentchat\groupchat.py", line 158, in select_speaker
final, name = selector.generate_oai_reply(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\asset\anaconda3\Lib\site-packages\autogen\agentchat\conversable_agent.py", line 625, in generate_oai_reply
response = client.create(
^^^^^^^^^^^^^^
File "C:\Users\asset\anaconda3\Lib\site-packages\autogen\oai\client.py", line 247, in create
response = self._completions_create(client, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\asset\anaconda3\Lib\site-packages\autogen\oai\client.py", line 327, in _completions_create
response = completions.create(**params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\asset\anaconda3\Lib\site-packages\openai\_utils\_utils.py", line 301, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\asset\anaconda3\Lib\site-packages\openai\resources\chat\completions.py", line 598, in create
return self._post(
^^^^^^^^^^^
File "C:\Users\asset\anaconda3\Lib\site-packages\openai\_base_client.py", line 1096, in post
return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\asset\anaconda3\Lib\site-packages\openai\_base_client.py", line 856, in request
return self._request(
^^^^^^^^^^^^^^
File "C:\Users\asset\anaconda3\Lib\site-packages\openai\_base_client.py", line 908, in _request
raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'error': "'messages' array must only contain objects with a 'content' field that is not empty."}@Maxime-Charmasson well it (the link) works for me, you can find this under Discord for LM studio feature request / support open ai 1.0 API
shuther
commented
8 months ago I understand it is expected to work better with GPT4, but is it possible to confirm how the next player should be written (with escape characters, ...). One "bad" trick could be to match it using fuzzy matching, so we go around little change (such as plurial/singular...), but I am still confused as the next player looks ok for me and we still have an issue to invalidate dirty cache?
afourney
commented
8 months ago Thanks for the additional information.
GroupChat is already doing a somewhat fuzzy match by counting how many times each agent is named in response, regardless of formatting: https://github.com/microsoft/autogen/blob/30daa1eac52f8d19f84df5c67cccbab0f9386463/autogen/agentchat/groupchat.py#L267C4-L288C24
It's possible there is a bug here, but I'd love to see the inputs and output to replicate.
As for the cache, the cache key should already consider the llm_config settings, so I'm not sure what's going on there. Again, it is possible that there is a bug with that, but it would impact things more broadly... not just group chat: https://github.com/microsoft/autogen/blob/30daa1eac52f8d19f84df5c67cccbab0f9386463/autogen/oai/openai_utils.py#L52-L71C46
I'll keep digging, but the biggest concern I'm seeing is that the speaker selection pre-prompt is missing several agent descriptions in the trace above. However, the system_prompts/descriptions don't match the configuration that was later provided, so something doesn't quite line up.
Ideally, what I would like to see is the configuration + the selection pre-prompts of the same run. We can then determine if there is a bug there.
The pre-prompt descriptions are generated here: https://github.com/microsoft/autogen/blob/30daa1eac52f8d19f84df5c67cccbab0f9386463/autogen/agentchat/groupchat.py#L253-L265
AssetDK
commented
8 months ago For starters I think i found the reason this does not work, but I may also have some other basic problem.
We often get this GroupChat select_speaker failed to resolve the next speaker's name. This is because the speaker selection OAI call returned:
This is how the code should actually run, but in some cases the director did not yield such a message.
I dont know it this is the root cause but i did not set the names correct:
The other thing is that I would think that searching the agents names/roles may be corrupted or "mixed up" when the name is actually a role - that is you should perhaps call the Director agent "joe" - and we should not use the common terms "Director", "Assistant", "Programmer" etc. - even "DirectorA" will be better... NLP is not easy to process programmatically.
This is debug from groupchat.py
Sorry for the unfinnised message got to run
afourney
commented
8 months ago @AssetDK thanks again for working through this. I really do believe that a good chunk of the problem here is that the core prompting is tuned to GPT-4, and that's breaking speaker selection. The LLM is clearly not following even the basic formatting instructions for speaker selection (but we should be robust to simple formatting issues).
One thing that may help is adding a description field to each of your agents. Keep the descriptions short, sweet, and 3rd person. More on descriptions here: https://microsoft.github.io/autogen/blog/2023/12/29/AgentDescriptions
AssetDK
commented
8 months ago Thanks a lot Adam I have to read all those blog posts! I clearly did not look to much into the documentation of UserProxAgent and AssistantAgent.
I will use this for some specific ideas for making the agent produce higher quality output in different ways. I don't know yet if that can be implemented as finetuning or in some way of training. For prof of concept it will have to be done via memory, description or may be even via the prompt.
As an example I might need to "make" an expert in a specific branch of philosophy, but I would find it hard to keep that short and sweet... especially if I wanted to have to describe several such experts in distinct ways... that could evolve to long and complicated descriptions.
Also I need much more knowledge about setting up the environment, I have no idea about what versions I'm running on for any of the software - line numbers are deferent between our code, so we may not be running on the same versions.
I just now found this tutorial that is explaining the general setup for VS code (and teachable agents! ... something I will probably need for my project. https://www.youtube.com/watch?v=KDpGN7QDEVk (Autogen 0.2.0 where as your talking about version 0.2.2 )
Finally, it looks like the version I am running on uses a seed (see below) to preserve some kind of memory I got no idea how that works and how to reset? (teachable agents sounds better, so i will probably need to test that out!)
I wonder if Teachable agent is supplementing or replacing memgpt? - and all of this should probably be in the diskussions..
config = {
"config_list": [
{
....,
....
}
],
"seed": 1114
}
rickyloynd-microsoft
commented
8 months ago I wonder if Teachable agent is supplementing or replacing memgpt?
Autogen's teachable agents are related to MemGPT in using a vector db for memory that gets constantly updated. But their details and capabilities are quite different.
AssetDK
commented
8 months ago @afourney - When I disregard the original problem it looks like things are working "somewhat"... until I get this Trackback message, I guess that the code could be fragile if it depends on the reply syntax and try to convert in to messages.
The description parameter works fine, however I needed to fill out the system_message parameter for my "Director" and looks like all of the agents are running under a 'user' role even my "Director" - I was under the impression that UserProxyAgent would be the mannager of what agent was next.
... and I still get the GroupChat select_speaker failed to resolve the next speaker's name. This is because the speaker selection OAI call returned: message... and that seems to keep the "team" in a kind of loop! (I will try to debug that.
Also I will make a new discussion regarding how the team set-up is working. Agent Descriptions (version 0.2.2 onwards) and team setup. #1109
Last message: { role: 'user', content: ' I have provided a detailed response addressing your questions regarding creating a team of `autogen AssistantAgent` agents for EU's Common Agricultural Policy (CAP). With these guidelines, you should be able to develop a capable and efficient team of assistant agents that can assist users with any queries they may have about CAP legislation.
If you have any further questions or need additional clarification, don't hesitate to ask!
TERMINATE' } Step 3: Develop the AssistantAgent agents
Now that we have defined the tasks and skills for our AssistantAgent team, we can start developing them. We will write Python code that uses natural language processing (NLP) techniques to understand user queries and provide accurate answers based on their knowledge of EU's CAP legislation. We will also implement a system for updating our agents with any changes in the CAP regulations as they occur.
Plan: Develop the AssistantAgent team using Python, integrating NLP libraries such as spaCy or NLTK and databases to store and access CAP information.
# filename: assistant_agents.py
python -m venv cap_assistant_venv
source cap_assistant_venv/bin/activate
pip install spacy nltk sqlite3
git clone https://github.com/explosion/spaCy.git
cd spaCy && python setup.py install --user
deactivate
ithllc
commented
8 months ago First of all, Happy New Year to everyone, and adding my 2 cents in here as well. I would like to the authors at Microsoft and Seehiong for posting workable examples. I too received the same thing but everything else works fine as for the group chat conversation from what I can see so far and I cannot get the user_proxy to execute code in WSL2 Ubuntu 22.04 but I don't think that is related to this issue at all. I agree with @afourney that it's the dependency built into the OpenAI GPT LLMs but why should we have to to also teach the agents who the agents are when in my code, it clearly lists the names of the agents, which like @AssetDK says, is also the roles as well? If that is the case, then how to do we implement teachability to the group chat manager immediately after builder builds all the agents, because based on the code below, they start talking as soon as the function "start_task" completes. This is 100% reproducible from the code below. Thank you, Seehiong, for the great tutorial. Here's my setup and code below:
windows 11 LM Studio version: 2.10 for windows Local LLM: Mistral Instruct v2 7B
WSL2 Ubuntu 22.04 Python 3.10 pyautogen 0.2.2
Code base:
import autogen
from autogen.agentchat.contrib.agent_builder import AgentBuilder
# https://seehiong.github.io/2023/unveiling-agent-autobuild-in-autogen/
config_path = 'OAI_CONFIG_LIST'
default_llm_config = {
"temperature": 0,
'timeout': 600
}
builder = AgentBuilder(config_path=config_path, builder_model='gpt-4', agent_model='gpt-4')
custom_task = """
I'm enthusiastic about launching an innovative project â a software academy tailored for children aged 7 to 16 with no prior coding experience.
The focus is on AI and introductory Python programming to impart essential technical skills crucial for their future.
To bring this vision to life, I'd like the language model's assistance in detailing the course agenda, developing a comprehensive curriculum, and establishing a step-by-step approach towards our goal.
Together, we aim to create a compelling plan that not only educates but also inspires the next generation of tech enthusiasts.
Let's embark on this exciting journey by shaping the foundation of our software academy.
"""
agent_list, agent_configs = builder.build(custom_task + '\nOnly identify the agent name and separate them with comma.', default_llm_config)
def start_task(execution_task: str, agent_list: list, llm_config: dict):
config_list = autogen.config_list_from_json(config_path, filter_dict={"model": ["gpt-4"]})
group_chat = autogen.GroupChat(agents=agent_list, messages=[], max_round=12)
manager = autogen.GroupChatManager(
groupchat=group_chat, llm_config={"config_list": config_list, **llm_config}
)
agent_list[0].initiate_chat(manager, message=execution_task)
start_task(
execution_task=custom_task,
agent_list=agent_list,
llm_config=default_llm_config
)
netixc
commented
8 months ago im not sure if this is the correct way to solve the problem but i use the following and have no problems
manager = autogen.GroupChatManager(groupchat=groupchat, llm_config=llm_config)```
only thing i did was adding the ```speaker_selection_method="round_robin```Round robin will definitely solve it, but then the order is fixed -- it just goes round and round in the order you specified. Is that what you want for your scenario?
ithllc
commented
8 months ago Providing an update, I was able to get it to work using "speaker_selection_method", the only one that did not work was for "auto".
group_chat = autogen.GroupChat(agents=agent_list, messages=[], max_round=12,speaker_selection_method="manual")
for the code:
speaker_selection_method: the method for selecting the next speaker. Default is "auto". Could be any of the following (case insensitive), will raise ValueError if not recognized:
"auto": the next speaker is selected automatically by LLM.
"manual": the next speaker is selected manually by user input.
"random": the next speaker is selected randomly.
"round_robin": the next speaker is selected in a round robin fashion, i.e., iterating in the same order as provided in agents.
matsuobasho
commented
8 months ago I think this may be a different issue, but I get the same error as @AssetDK replicating the code generation tutorial.
Error code: 400 - {'error': "'messages' array must only contain objects with a 'content' field that is not empty."}
Running the following with llama2 13B:
import autogen
config_list = [
{
"base_url": "http://localhost:1234/v1",
"api_key": "NULL"
}
]
llm_config = {
"timeout": 600,
"seed": 40,
"config_list": config_list,
"temperature": 0
}
# create an AssistantAgent named "assistant"
assistant = autogen.AssistantAgent(
name="assistant",
llm_config={
"cache_seed": 55, # seed for caching and reproducibility
"config_list": config_list, # a list of OpenAI API configurations
"temperature": 0, # temperature for sampling
}, # configuration for autogen's enhanced inference API which is compatible with OpenAI API
)
# create a UserProxyAgent instance named "user_proxy"
user_proxy = autogen.UserProxyAgent(
name="user_proxy",
human_input_mode="NEVER",
max_consecutive_auto_reply=10,
is_termination_msg=lambda x: x.get("content", "").rstrip().endswith("TERMINATE"),
code_execution_config={
"work_dir": "coding",
"use_docker": False, # set to True or image name like "python:3" to use docker
},
)
# the assistant receives a message from the user_proxy, which contains the task description
user_proxy.initiate_chat(
assistant,
message="""What date is today? Compare the year-to-date gain for META and TESLA.""",
)Can open separate issue for this if hasn't been addressed here.
afourney
commented
8 months ago @matsuobasho I've seen this error mentioned a few times. E.g., https://github.com/microsoft/autogen/issues/279#issuecomment-1773528968
If you share a trace we can debug further.
matsuobasho
commented
8 months ago @afourney , sure here is the traceback:
BadRequestError Traceback (most recent call last)
Cell In[3], line 22
11 user_proxy = autogen.UserProxyAgent(
12 name="user_proxy",
13 human_input_mode="NEVER",
(...)
19 },
20 )
21 # the assistant receives a message from the user_proxy, which contains the task description
---> 22 user_proxy.initiate_chat(
23 assistant,
24 message="""What date is today? Compare the year-to-date gain for META and TESLA.""",
25 )
File c:\Users\Bob\.virtualenvs\autogen\Lib\site-packages\autogen\agentchat\conversable_agent.py:556, in ConversableAgent.initiate_chat(self, recipient, clear_history, silent, **context)
542 """Initiate a chat with the recipient agent.
543
544 Reset the consecutive auto reply counter.
(...)
553 "message" needs to be provided if the `generate_init_message` method is not overridden.
554 """
555 self._prepare_chat(recipient, clear_history)
--> 556 self.send(self.generate_init_message(**context), recipient, silent=silent)
File c:\Users\Bob\.virtualenvs\autogen\Lib\site-packages\autogen\agentchat\conversable_agent.py:354, in ConversableAgent.send(self, message, recipient, request_reply, silent)
352 valid = self._append_oai_message(message, "assistant", recipient)
353 if valid:
--> 354 recipient.receive(message, self, request_reply, silent)
355 else:
356 raise ValueError(
357 "Message can't be converted into a valid ChatCompletion message. Either content or function_call must be provided."
358 )
File c:\Users\Bob\.virtualenvs\autogen\Lib\site-packages\autogen\agentchat\conversable_agent.py:489, in ConversableAgent.receive(self, message, sender, request_reply, silent)
487 reply = self.generate_reply(messages=self.chat_messages[sender], sender=sender)
488 if reply is not None:
--> 489 self.send(reply, sender, silent=silent)
File c:\Users\Bob\.virtualenvs\autogen\Lib\site-packages\autogen\agentchat\conversable_agent.py:354, in ConversableAgent.send(self, message, recipient, request_reply, silent)
352 valid = self._append_oai_message(message, "assistant", recipient)
353 if valid:
--> 354 recipient.receive(message, self, request_reply, silent)
355 else:
356 raise ValueError(
357 "Message can't be converted into a valid ChatCompletion message. Either content or function_call must be provided."
358 )
File c:\Users\Bob\.virtualenvs\autogen\Lib\site-packages\autogen\agentchat\conversable_agent.py:489, in ConversableAgent.receive(self, message, sender, request_reply, silent)
487 reply = self.generate_reply(messages=self.chat_messages[sender], sender=sender)
488 if reply is not None:
--> 489 self.send(reply, sender, silent=silent)
File c:\Users\Bob\.virtualenvs\autogen\Lib\site-packages\autogen\agentchat\conversable_agent.py:354, in ConversableAgent.send(self, message, recipient, request_reply, silent)
352 valid = self._append_oai_message(message, "assistant", recipient)
353 if valid:
--> 354 recipient.receive(message, self, request_reply, silent)
355 else:
356 raise ValueError(
357 "Message can't be converted into a valid ChatCompletion message. Either content or function_call must be provided."
358 )
File c:\Users\Bob\.virtualenvs\autogen\Lib\site-packages\autogen\agentchat\conversable_agent.py:487, in ConversableAgent.receive(self, message, sender, request_reply, silent)
485 if request_reply is False or request_reply is None and self.reply_at_receive[sender] is False:
486 return
--> 487 reply = self.generate_reply(messages=self.chat_messages[sender], sender=sender)
488 if reply is not None:
489 self.send(reply, sender, silent=silent)
File c:\Users\Bob\.virtualenvs\autogen\Lib\site-packages\autogen\agentchat\conversable_agent.py:962, in ConversableAgent.generate_reply(self, messages, sender, exclude)
960 continue
961 if self._match_trigger(reply_func_tuple["trigger"], sender):
--> 962 final, reply = reply_func(self, messages=messages, sender=sender, config=reply_func_tuple["config"])
963 if final:
964 return reply
File c:\Users\Bob\.virtualenvs\autogen\Lib\site-packages\autogen\agentchat\conversable_agent.py:631, in ConversableAgent.generate_oai_reply(self, messages, sender, config)
628 messages = self._oai_messages[sender]
630 # TODO: #1143 handle token limit exceeded error
--> 631 response = client.create(
632 context=messages[-1].pop("context", None), messages=self._oai_system_message + messages
633 )
635 # TODO: line 301, line 271 is converting messages to dict. Can be removed after ChatCompletionMessage_to_dict is merged.
636 extracted_response = client.extract_text_or_completion_object(response)[0]
File c:\Users\Bob\.virtualenvs\autogen\Lib\site-packages\autogen\oai\client.py:260, in OpenAIWrapper.create(self, **config)
258 continue # filter is not passed; try the next config
259 try:
--> 260 response = self._completions_create(client, params)
261 except APIError as err:
262 error_code = getattr(err, "code", None)
File c:\Users\Bob\.virtualenvs\autogen\Lib\site-packages\autogen\oai\client.py:346, in OpenAIWrapper._completions_create(self, client, params)
344 params = params.copy()
345 params["stream"] = False
--> 346 response = completions.create(**params)
347 return response
File c:\Users\Bob\.virtualenvs\autogen\Lib\site-packages\openai\_utils\_utils.py:272, in required_args.<locals>.inner.<locals>.wrapper(*args, **kwargs)
270 msg = f"Missing required argument: {quote(missing[0])}"
271 raise TypeError(msg)
--> 272 return func(*args, **kwargs)
File c:\Users\Bob\.virtualenvs\autogen\Lib\site-packages\openai\resources\chat\completions.py:645, in Completions.create(self, messages, model, frequency_penalty, function_call, functions, logit_bias, logprobs, max_tokens, n, presence_penalty, response_format, seed, stop, stream, temperature, tool_choice, tools, top_logprobs, top_p, user, extra_headers, extra_query, extra_body, timeout)
596 @required_args(["messages", "model"], ["messages", "model", "stream"])
597 def create(
598 self,
(...)
643 timeout: float | httpx.Timeout | None | NotGiven = NOT_GIVEN,
644 ) -> ChatCompletion | Stream[ChatCompletionChunk]:
--> 645 return self._post(
646 "/chat/completions",
647 body=maybe_transform(
648 {
649 "messages": messages,
650 "model": model,
651 "frequency_penalty": frequency_penalty,
652 "function_call": function_call,
653 "functions": functions,
654 "logit_bias": logit_bias,
655 "logprobs": logprobs,
656 "max_tokens": max_tokens,
657 "n": n,
658 "presence_penalty": presence_penalty,
659 "response_format": response_format,
660 "seed": seed,
661 "stop": stop,
662 "stream": stream,
663 "temperature": temperature,
664 "tool_choice": tool_choice,
665 "tools": tools,
666 "top_logprobs": top_logprobs,
667 "top_p": top_p,
668 "user": user,
669 },
670 completion_create_params.CompletionCreateParams,
671 ),
672 options=make_request_options(
673 extra_headers=extra_headers, extra_query=extra_query, extra_body=extra_body, timeout=timeout
674 ),
675 cast_to=ChatCompletion,
676 stream=stream or False,
677 stream_cls=Stream[ChatCompletionChunk],
678 )
File c:\Users\Bob\.virtualenvs\autogen\Lib\site-packages\openai\_base_client.py:1088, in SyncAPIClient.post(self, path, cast_to, body, options, files, stream, stream_cls)
1074 def post(
1075 self,
1076 path: str,
(...)
1083 stream_cls: type[_StreamT] | None = None,
1084 ) -> ResponseT | _StreamT:
1085 opts = FinalRequestOptions.construct(
1086 method="post", url=path, json_data=body, files=to_httpx_files(files), **options
1087 )
-> 1088 return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
File c:\Users\Bob\.virtualenvs\autogen\Lib\site-packages\openai\_base_client.py:853, in SyncAPIClient.request(self, cast_to, options, remaining_retries, stream, stream_cls)
844 def request(
845 self,
846 cast_to: Type[ResponseT],
(...)
851 stream_cls: type[_StreamT] | None = None,
852 ) -> ResponseT | _StreamT:
--> 853 return self._request(
854 cast_to=cast_to,
855 options=options,
856 stream=stream,
857 stream_cls=stream_cls,
858 remaining_retries=remaining_retries,
859 )
File c:\Users\Bob\.virtualenvs\autogen\Lib\site-packages\openai\_base_client.py:930, in SyncAPIClient._request(self, cast_to, options, remaining_retries, stream, stream_cls)
927 if not err.response.is_closed:
928 err.response.read()
--> 930 raise self._make_status_error_from_response(err.response) from None
932 return self._process_response(
933 cast_to=cast_to,
934 options=options,
(...)
937 stream_cls=stream_cls,
938 )
BadRequestError: Error code: 400 - {'error': "'messages' array must only contain objects with a 'content' field that is not empty."}@afourney Version 0.2.3 of pyautogen I finally got a bit of time and found that this is what is happening when the no 'content' message occurs.
To me it looks like the message = messages[-1] - statement produces an empty message... where it is supposed to return the last message. I am not 100% sure about how Python should handle this... but it looks like the two elements in the list have the values in different sequences. I do not know if that is causing the problem and where t fix it?
A simple temporary solution would be to simply do this for groupchat.py line 342:
if message["content"] != "": groupchat.append(message)
An empty message will be sent to the other agents - so for that I made the same exception:
This does not solve the original problem where we do not get the next speaker... but the chat now runs and runs!
j4ys0n
commented
5 months ago has anyone figured out what the problem or solution is here? it's definitely speaker selection, but it's not clear to me why it's happening.
AndreiD
commented
5 months ago project was abandoned. they did some hype to get some funding after that ... fixing styling and typing errors :)) no one gives a shit about bugs
ekzhu
commented
5 months ago @j4ys0n @AndreiD
If the issue is regarding to selecting speaker due to the LLM adding "\" to "_", this should have been fixed by: https://github.com/microsoft/autogen/pull/2267.
If it is a different issue, please create a new one with a clear description of what the issue is.
ekzhu
commented
5 months ago Closing this as https://github.com/microsoft/autogen/pull/2267 has fixed the issue described in the issue description.
fengzeyuchen
commented
4 months ago {
"building_task": "..."
"agent_configs": [
{
"name": "Fiction_Knowledge_Collector",
"model": "gpt-4",
"system_message": "You are now in a group chat. You need to complete a task with other participants. As a Fiction_Knowledge_Collector, your duty is to:\n\n- Collect and analyze information related to novelists, screenwriters, and film critics.\n- Provide all the necessary information to write a complete novel.\n- Ensure the constructed worldview, values, and marital values are realistic and logically coherent.\n- Provide comprehensive and complete evaluations, including plot construction, characterization, scene description, emotional rendering, and knowledge.\n\nTo complete the task, follow these steps:\n\n1. Collect information about novelists, screenwriters, and film critics. Use the following Python code to browse the web and search for relevant information:\n\n # filename: browse_web.py\n import webbrowser\n url = 'https://www.google.com/search?q=novelists+screenwriters+film+critics'\n webbrowser.open(url)\n\n2. Analyze the collected information to identify the necessary elements for writing a novel.\n\n3. Provide the collected information and analysis to the other participants in the group chat.\n\n4. Answer any questions that the other participants may have.\n\n5. Continue to collect and analyze information as needed until the task is complete.\n\nPlease let me know when you are finished with the task by replying \"TERMINATE\" in the group chat.",
"description": "Fiction_Knowledge_Collector is a knowledgeable person who understands the ins and outs of most works of fiction. They have the ability to accurately recall and organize information from a variety of fictional sources, including books, movies, TV shows, and video games. Fiction_Knowledge_Collectors should be able to use their knowledge to answer questions about the plot, characters, or settings of these works."
},
{
"name": "Script_Knowledge_Collector",
"model": "gpt-4",
"system_message": "You are now in a group chat. You need to complete a task with other participants. As a Script_Knowledge_Collector, you need to modify the following position requirement, making it more suitable for the above task and position:\n\n```\nYou are a helpful AI assistant.\nSolve tasks using your coding and language skills.\nIn the following cases, suggest python code (in a python coding block) or shell script (in a sh coding block) for the user to execute.\n 1. When you need to collect info, use the code to output the info you need, for example, browse or search the web, download/read a file, print the content of a webpage or a file, get the current date/time, check the operating system. After sufficient info is printed and the task is ready to be solved based on your language skill, you can solve the task by yourself.\n 2. When you need to perform some task with code, use the code to perform the task and output the result. Finish the task smartly.\nSolve the task step by step if you need to. If a plan is not provided, explain your plan first. Be clear which step uses code, and which step uses your language skill.\nWhen using code, you must indicate the script type in the code block. The user cannot provide any other feedback or perform any other action beyond executing the code you suggest. The user can't modify your code. So do not suggest incomplete code which requires users to modify. Don't use a code block if it's not intended to be executed by the user.\nIf you want the user to save the code in a file before executing it, put # filename: <filename> inside the code block as the first line. Don't include multiple code blocks in one response. Do not ask users to copy and paste the result. Instead, use 'print' function for the output when relevant. Check the execution result returned by the user.\nIf the result indicates there is an error, fix the error and output the code again. Suggest the full code instead of partial code or code changes. If the error can't be fixed or if the task is not solved even after the code is executed successfully, analyze the problem, revisit your assumption, collect additional info you need, and think of a different approach to try.\nWhen you find an answer, verify the answer carefully. Include verifiable evidence in your response if possible.\nReply \"TERMINATE\" in the end when everything is done.\n```\n\nHint:\n\n# Your answer should be natural, starting from \"You are now in a group chat. You need to complete a task with other participants. As a ...\".\n# [IMPORTANT] You should let them reply \"TERMINATE\" when they think the task is completed (the user's need has actually been satisfied).\n# The modified requirement should not contain the code interpreter skill.\n# You should remove the related skill description when the position is not a programmer or developer.\n# Coding skill is limited to Python.\n# Your answer should omit the word \"REQUIREMENT\".\n# People with the above position can doubt previous messages or code in the group chat (for example, if there is no\noutput after executing the code) and provide a corrected answer or code.\n# People in the above position should ask for help from the group chat manager when confused and let the manager select another participant.",
"description": "Script Knowledge Collector is a position for people who are familiar with scripts or natural languages. They can identify the errors in the scripts or natural languages and provide corrected answers or code. They should have good analytical skills and be able to think critically."
},
{
"name": "Film_Review_Collector",
"model": "gpt-4",
"system_message": "You are now in a group chat. You need to complete a task with other participants. As a Film_Review_Collector, you need to focus on the following aspects:\n\n1. **Novel writing:** You should provide important information and evidence for the construction of a worldview and provide a reasonable and logical closed loop for values and views on marriage and love.\n\n2. **Film review:** You need to provide a comprehensive and complete evaluation of the film, including but not limited to plot construction, character shaping, scene description, and emotional rendering.\n\nPlease note that you should not provide code or shell scripts for the user to execute. Your responses should be natural language only. When the task is completed, please reply \"TERMINATE\" in the chat.",
"description": "Film_Review_Collector is a senior member who is experienced in film or TV series reviewing and writing. This person can provide high-quality film/series reviews for films/series they watched."
},
{
"name": "Fiction_Analysis_Expert",
"model": "gpt-4",
"system_message": "You are now in a group chat. You need to complete a task with other participants. As a Fiction_Analysis_Expert, you are responsible for generating all necessary information to help write a complete novel, including collecting and analyzing specific areas relevant to fiction writers, screenwriters, and film critics. This information will serve as a solid foundation for constructing a coherent and realistic worldview, value system, and view on marriage and love in your novel. It will also provide comprehensive and thorough guidance on all aspects of novel writing, including plot construction, character development, scene description, emotional rendering, knowledge supplementation, and expansion. Your work should be of high quality and include appropriate evaluation and optimization based on film reviews. You can ask for assistance from the chat manager if you have any questions. Let us know when you are done by replying \"TERMINATE.\"",
"description": "Fiction_Analysis_Expert is a position reserved for individuals who have studied fiction, literature, and other works of prose. They have an encyclopedic knowledge of these works, including authors, themes, styles, and character analyses. Individuals with this position should have experience in writing, editing, and presenting their findings on literary topics."
},
{
"name": "Fiction_Quality_Optimizer",
"model": "gpt-4",
"system_message": "You are now in a group chat. You need to complete a task with other participants. As a Fiction Quality Optimizer, you are responsible for ensuring the quality of fiction writing. This includes collecting and analyzing relevant information, providing feedback on writing, and helping writers improve their craft.\n\nBased on the current task and your position, the following requirement is more suitable:\n\nYou are a helpful AI assistant.\nSolve tasks using your research, analytical, and communication skills on the following aspects:\n\n1. Information gathering and analysis: collect and analyze information on topics related to fiction writing, such as character development, plot structure, worldbuilding, and literary techniques.\n2. Feedback and optimization: provide feedback on fiction writing, including plot structure, character development, worldbuilding, and literary techniques. Suggest improvements and optimizations to enhance the quality of the writing.\n3. Knowledge expansion: supplement and expand the writer's knowledge on fiction writing techniques and best practices through the provision of relevant resources and insights.\n\nIn the following cases, provide guidance on research, analysis, and communication strategies for the user to execute:\n\n1. When you need to collect info, guide the user on how to conduct research on the relevant topic, using search engines, online databases, or other resources.\n2. When you need to provide feedback and optimization, guide the user on how to analyze the writing and identify areas for improvement, using specific examples and explanations.\n3. When you need to expand the writer's knowledge, guide the user on how to access and utilize resources such as articles, books, workshops, or online courses.\n\nSolve the task step by step if you need to. If a plan is not provided, explain your plan first. Be clear which step uses research and analysis, and which step uses communication skills.\n\nWhen using code, you must indicate the script type in the code block. The user cannot provide any other feedback or perform any other action beyond executing the code you suggest. The user can't modify your code. So do not suggest incomplete code which requires users to modify. Don't use a code block if it's not intended to be executed by the user.\n\nIf you want the user to save the code in a file before executing it, put # filename: <filename> inside the code block as the first line. Don't include multiple code blocks in one response. Do not ask users to copy and paste the result. Instead, use 'print' function for the output when relevant. Check the execution result returned by the user.\n\nIf the result indicates there is an error, fix the error and output the code again. Suggest the full code instead of partial code or code changes. If the error can't be fixed or if the task is not solved even after the code is executed successfully, analyze the problem, revisit your assumption, collect additional info you need, and think of a different approach to try.\n\nWhen you find an answer, verify the answer carefully. Include verifiable evidence in your response if possible.\n\nReply \"TERMINATE\" in the end when everything is done.",
"description": "Fiction_Quality_Optimizer is a role that requires intellectual curiosity, meticulous proofreading, an eye for detail, and a keen grasp of storytelling principles. Their responsibilities include identifying inconsistencies or inaccuracies in content, assessing narrative structure and character development, and ensuring that the overall quality of the fiction meets the highest standards."
}
],
"coding": false,
"default_llm_config": {
"temperature": 0,
"timeout": 300
},
"code_execution_config": {
"last_n_messages": 2,
"work_dir": "groupchat",
"use_docker": false,
"timeout": 60
}
}
The above is the Agent configuration generated by buderagent, including the description, but the process is not satisfactory, and it still fails to match when executing tasks, and I do not know how to work properly. In addition, it seems that there are only five agents at most when Builder creates agents. No matter how much description is given to it, it still generates 5 agents so far, and here is a look at the task and the execution error:
start_task(
execution_task="Began to write a novel, the theme is to cultivate the immortal, the protagonist is called Jiang Xin",
agent_list=agent_list
)Output:
Fiction_Knowledge_Collector (to chat_manager):
Began to write a novel, the theme is to cultivate the immortal, the protagonist is called Jiang Xin
--------------------------------------------------------------------------------
WARNING:autogen.agentchat.groupchat:GroupChat select_speaker failed to resolve the next speaker's name. This is because the speaker selection OAI call returned:
Script_Knowledge_Collector (to chat_manager):
You are a well-informed, all-knowing novelist.
Please write a novel about Jiang Xin's cultivation into a fairy. The novel should be lively and interesting. The novel should include Jiang Xin's growth, the challenges he encountered, and the process of eventually becoming an immortal.
Please reply once each section is completed for review.
At the end of each response, please add a short summary of Jiang Xin's current stage.
Please do not reply "Finished" until the novel is finished.
In your final reply, please END with "end" to indicate that the novel is complete.
--------------------------------------------------------------------------------
WARNING:autogen.agentchat.groupchat:GroupChat select_speaker failed to resolve the next speaker's name. This is because the speaker selection OAI call returned:
---------------------------------------------------------------------------
BadRequestError Traceback (most recent call last)
[<ipython-input-15-80357df72228>](https://localhost:8080/#) in <cell line: 1>()
----> 1 start_task(
2 execution_task="Began to write a novel, the theme is to cultivate the immortal, the protagonist is called Jiang Xin",
3 agent_list=agent_list
4 )
15 frames
[/usr/local/lib/python3.10/dist-packages/openai/_base_client.py](https://localhost:8080/#) in _request(self, cast_to, options, remaining_retries, stream, stream_cls)
1011
1012 log.debug("Re-raising status error")
-> 1013 raise self._make_status_error_from_response(err.response) from None
1014
1015 return self._process_response(
BadRequestError: Error code: 400 - {'error': {'message': 'Please ensure that multiturn requests ends with a user role or a function response. (request id: 2024042217473221988766718854451)', 'type': '', 'param': '', 'code': 400}}
marklysze
commented
4 months ago @fengzeyuchen, are you able to create a new issue as it's not specifically for this issue, and also provide your full code?
could be connected to #663. I am using the latest released version (0.2) I am getting error such as:
while the role is: Senior_Python_Engineer
Note sure at which stage the space character gets converted to _.