Open husseinmozannar opened 10 months ago
Fantastic bug report @husseinmozannar (and hello!)
cc @victordibia and @gagb who I believe worked on that notebook.
Let's do some test with it. I tried this before and it degraded the performance in some cases (e.g., when the output is long). It could be potentially improved with the idea of #439 though.
Bug in notebook example:
There is an issue in a notebook example (and one thing that can improve it) https://github.com/microsoft/autogen/blob/main/notebook/agentchat_groupchat_vis.ipynb
TL;DR: Coder ends up downloading dataset from a different URL (without verifying that it is the same!) instead of debugging and thus the task is deemed complete. This occurs because the User_proxy execution does not provide the partial execution trace (output of the print(df.columns) statement) and thus the coder does not know how to debug.
Recommendation: Add partial execution results when user_proxy talks to chat_manager
Potential fix: on main branch as of 10/25/2023 10am ET, update autogen/code_utils.py line 326-335
to:
I haven't tested this properly, but think it might work.
At some point, the coder has the code
this code won't run because "Weight_in_lbs" is not a column in the df, BUT if you had provided the partial execution trace to the agent which would include the names of the columns:
it would easily figure it out! BUT the user_proxy does not give it this information! only that an exception happened that "Weight_in_lbs" is not valid.
The conversation becomes repetitive for 2-3 rounds with the same error, effectively stuck in a loop.
Until the critic says:
Which is an invalid URL that the coding agent has to change, thus marking clear break in communication!
The invalid URL is then changed to
url = "https://raw.githubusercontent.com/vega/vega-datasets/main/data/cars.json"
and the code runs and outputs a nice plot. But the user has no idea that the agents completely changed the correct URL.