[Bug]: General Agent Workflow gets stuck in its own conversation and never answer the question.

AKRFranko commented 9 months ago

Describe the bug

I asked the general agent workflow "can you help me program a new autogenstudio skill?"

and eventually got the response:

"As there seems to be a persistent issue with receiving your messages, I recommend checking your device or messaging platform for any technical problems that might be preventing your messages from being sent. If you need assistance at a later time, please don't hesitate to reach out. I'm here to help whenever you're ready. Have a great day! TERMINATE"

The problem is the assistant asks for more details (which is not bubbled up to the UI) and the userproxy does not respond so eventually the conversation drops ie:

userproxy (to primary_assistant):

can you help me program a new autogenstudio skill

--------------------------------------------------------------------------------
primary_assistant (to userproxy):

Certainly! To program a new skill for autogenstudio, we'll need to define the functionality you want to create, the inputs and outputs, and any specific requirements or constraints. Please provide me with the details of the skill you'd like to develop, and I'll assist you with the coding part.

--------------------------------------------------------------------------------
userproxy (to primary_assistant):

--------------------------------------------------------------------------------
primary_assistant (to userproxy):

It seems like your message might have been cut off. Could you please provide the details of the skill you'd like to develop for autogenstudio? Specifically, I need to know the following:

1. The purpose of the skill: What task should it perform?
2. The inputs required: What information does the skill need to function?
3. The outputs expected: What should the skill produce or return?
4. Any specific technologies or APIs you want to use.
5. Any other requirements or constraints.

Once you provide this information, I can help you with the programming part.

--------------------------------------------------------------------------------
userproxy (to primary_assistant):

--------------------------------------------------------------------------------
primary_assistant (to userproxy):

It appears there might be an issue with your messages, as they are coming through empty. If you're trying to send details about the skill you want to develop and encountering issues, please try to resend the information. If you have any other requests or need assistance with a different task, please let me know so I can help you accordingly.

--------------------------------------------------------------------------------
userproxy (to primary_assistant):

--------------------------------------------------------------------------------
primary_assistant (to userproxy):

Since I am not receiving any specific details from you, I'm unable to proceed with programming a new autogenstudio skill. If you have any other requests or need assistance with a different task, please provide the relevant details, and I'll be glad to help you. If you're experiencing issues with sending messages, please check your connection or try refreshing the page. If there's nothing else you need at the moment, feel free to reach out whenever you're ready.

--------------------------------------------------------------------------------
userproxy (to primary_assistant):

--------------------------------------------------------------------------------
primary_assistant (to userproxy):

It seems there is still an issue with communication. If you are trying to send a message and it's not coming through, please check your connection or the platform you are using to ensure messages are being transmitted correctly. If you have a request or need assistance, please attempt to send your message again. If there is nothing further, you can reach out at any time when you're ready to proceed.

--------------------------------------------------------------------------------
userproxy (to primary_assistant):

--------------------------------------------------------------------------------
primary_assistant (to userproxy):

As there seems to be a persistent issue with receiving your messages, I recommend checking your device or messaging platform for any technical problems that might be preventing your messages from being sent. If you need assistance at a later time, please don't hesitate to reach out. I'm here to help whenever you're ready. Have a great day! TERMINATE

Steps to reproduce

ask it "can you help me program a new autogenstudio skill"

Expected Behavior

It should have asked me:

"Certainly! To program a new skill for autogenstudio, we'll need to define the functionality you want to create, the inputs and outputs, and any specific requirements or constraints. Please provide me with the details of the skill you'd like to develop, and I'll assist you with the coding part."

Screenshots and logs

No response

Additional Information

autgen version: 0.0.27a0 operating system: Linux DESKTOP-C9MELDE 5.15.133.1-microsoft-standard-WSL2 #1 SMP Thu Oct 5 21:02:42 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux python version: 3.10.12

ekzhu commented 9 months ago

Could you also post your OAI_CONFIG_LIST without the secrets?

AKRFranko commented 9 months ago

I have no idea where to find this. All i did was run:

$ export OPENAI_API_KEY=<my api key>
$ pipx install autogenstudio

I am running on Ubuntu in windows. I also tried activating autogenstudio's python venv and ran $ env, there is no evironment variable called "OAI_CONFIG_LIST" there.

ekzhu commented 9 months ago

Sorry I misunderstood your issue. It looks like the user proxy's human feedback setting is likely set to NEVER. Did you set this up in AutoGen Studio? Perhaps you can post a screenshot of your agent configurations.

AKRFranko commented 9 months ago

Thank you. This happened with the default out-of-the-box agent.

never agent

And i can't change it to anything else but "NEVER".

It works with other prompts, but some prompts have the same behavior as described above.

ekzhu commented 9 months ago

@victordibia

victordibia commented 9 months ago

Hi @AKRFranko ,

Currently, AutoGen studio only supports human input model NEVER. Supporting other modes is on the roadmap and requires additional streaming infrastructure. This is also discussed in #1278 .

Another factor that can cause agents to get stuck in a loop is related to the underlying LLM being used. In general larger models (GPT 3.5 and co) perform better and make less of these mistakes.

What model are you using?

AKRFranko commented 9 months ago

@victordibia I see.

I am using the default "general agent workflow" which has these models configured:

gpt-4-1106-preview gpt-3.5-turbo-16k TheBloke/zephyr-7B-alpha-AWQ

I will play around with models as you suggest.

Regarding this ticket, having explored a bit further, and understanding the semi-organic nature of this sort of cutting-edge AI tooling, I understand that just popping up the studio and expecting a perfect first run was a bit naive on my part.

I thank you very much for indulging me, keep up the great work, and i leave you the discretion of closing this ticket as you wish.

rysweet commented 1 week ago

not a bug

microsoft / autogen