[Issue]: Workflow terminates after 2 tokens when using AutogenStudio with LM Studio

[Freffles]

Describe the issue

If I create a model in Autogen studio that points to the LM studio endpoint then add the model to an agent, then a workflow etc, when I run the workflow it will terminate after 2 tokens. Works perfectly with Ollama but every time I try LM Studio I have this problem regardless of the model I try with LM Studio. This is the output from LM studio server log.

[2024-04-19 16:25:21.195] [INFO] [LM STUDIO SERVER] Processing queued request... [2024-04-19 16:25:21.197] [INFO] Received POST request to /v1/chat/completions with body: { "messages": [ { "content": "You are a helpful AI assistant", "role": "system" }, { "content": "what is the federal capital of australia", "role": "user" } ], "model": "gemma", "max_tokens": null, "stream": false, "temperature": 0.1 } [2024-04-19 16:25:21.198] [INFO] [LM STUDIO SERVER] Context Overflow Policy is: Rolling Window [2024-04-19 16:25:21.201] [INFO] [LM STUDIO SERVER] Last message: { role: 'user', content: 'what is the federal capital of australia' } (total messages = 2) [2024-04-19 16:25:21.696] [INFO] [LM STUDIO SERVER] Accumulating tokens ... (stream = false) [2024-04-19 16:25:21.698] [INFO] [lmstudio-ai/gemma-2b-it-GGUF/gemma-2b-it-q8_0.gguf] Accumulated 1 tokens: The [2024-04-19 16:25:21.799] [INFO] [lmstudio-ai/gemma-2b-it-GGUF/gemma-2b-it-q8_0.gguf] Accumulated 2 tokens: The Federal [2024-04-19 16:25:21.911] [INFO] [LM STUDIO SERVER] [lmstudio-ai/gemma-2b-it-GGUF/gemma-2b-it-q8_0.gguf] Generated prediction: { "id": "chatcmpl-zbpbxj8o70qatvsd8fz9ri", "object": "chat.completion", "created": 1713507921, "model": "lmstudio-ai/gemma-2b-it-GGUF/gemma-2b-it-q8_0.gguf", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "The Federal" }, "finish_reason": "stop" } ], "usage": { "prompt_tokens": 2, "completion_tokens": 2, "total_tokens": 4 } }

Steps to reproduce

I have no issue replicating on my setup. I have latest LM studio and Ollama. Anytime I create a model that is run on the LM Studio server I have this problem but not on the Ollama server

Screenshots and logs

No response

Additional Information

autogenstudio v0.0.56 LM Studio 0.2.19 with lmstudio-ai/gemma-2b-it-GGUF/gemma-2b-it-q8_0.gguf Ollama 0.1.32 with llama3:latest (8B)

[stigtk]

I have the exact same issue with lm online and autogen studio.

[castlenthesky]

I just encountered the same issue as well. My LM Studio logs are essentially identical to what you have shared.

[flynntmaverick]

Rolled back to 53 works - Same issue here. I'm able to get agents communicating with writing python,but autogen studio gets stuck with 2 tokens

[brustulim]

If anyone is interested in a "work-around" until the fix, you can: update the file: autogenstudio\datamodel.py by replacing the line 95 from: max_tokens: Optional[int] = None to: max_tokens: Optional[int] = 3000

then create a new model (exactly like your previous) and replace the old one in your agents, workflows, etc.

[ekzhu]

cc @victordibia for awareness

[victordibia]

@all, thanks for flagging this.

This has been fixed in the current dev branch for ags (pr here). In addition to a default value for max_tokens, we also expose the max_tokens parameter in the UI so users can specify max_tokens. You can try this branch out today by running pip install autogenstudio==0.0.56rc3.

Should be merged into main soon.

[Freffles]

@victordibia Thanks!

[Freffles]

replacing the line 95 from: max_tokens: Optional[int] = None to: max_tokens: Optional[int] = 3000

@brustulim Thanks for the idea. I was going to try something like that but I was not sure if it was the right solution.

[victordibia]

@godblessme

You seemed to have mentioned having issues with v0.0.56rc3 , and LM studio in a deleted comment ... can you verify this is the case?

[GodBlessingMe]

@godblessme

You seemed to have mentioned having issues with v0.0.56rc3 , and LM studio in a deleted comment ... can you verify this is the case?

@victordibia Thank you for providing the v0.0.56rc3 version. This version fixes my problem with the maximal limit of two tokens when using LM Studio. However, I encountered that Python code generated by primary-assistant cannot be run. Could you kindly help me figure out what happened? I provide the multi-conversations between userproxy and primary-assistant. Thank you very much.

[GodBlessingMe]

@victordibia Thank you. The issue was solved.

[MaxAkbar]

I'm using .NET code with LMStudio and it always fails on the second call. It works as expected when maxTokens is specified during agent creation.