[Bug]: Custom Model not supported in Group Chat with Speaker Prompt

cbrzn commented 3 months ago

Describe the bug

When trying to use Autogen only with Custom Models, the functionality of Group Chat with select_speaker_prompt_template attribute doesn't work as expected, because it creates a conversable agent which is not given the custom model client if exists

Steps to reproduce

1- Create GroupChatManager, which receives the LLM config from a local model, and use the register_model_client in the manager 2- Create multiple agents, with the LLM config from a local modal, and use the register_model_client in those 3- Create a Groupchat with select_speaker_prompt_template attribute defined, with the multiple agents 3- Create a user_proxy, which receives the LLM config from a local model and use the register_model_client in the user_proxy 4- Initiate chat between user_proxy and GroupChatManager, when it comes to select an agent it will throw this error because the ConversableAgent that defines the next agent (based on the speaker prompt template) doesn't have the local model

Model Used

https://huggingface.co/meetkai/functionary-small-v2.4-GGUF with LlamaCppPython

Expected Behavior

No response

Screenshots and logs

No response

Additional Information

No response

Hk669 commented 3 months ago

@cbrzn, we are enhancing Autogen support for non openai models, feel free to check the roadmap #2946 . thanks

wireless90 commented 2 months ago

Hi, i am facing a similar issue.

from typing import Any, List, Dict
from autogen import config_list_from_json, ConversableAgent, AssistantAgent, GroupChat, GroupChatManager

from neuroengine_client import NeuroengineClient
# Import configuration
llm_configuration = config_list_from_json("OAI_CONFIG_LIST.json", filter_dict={"model_client_cls": ["NeuroengineClient"]})
llm_configuration = {"config_list": llm_configuration}

# System Messages
ADDER_SYS_MESSAGE = "You will be given an integer, simply add one to it and return the result"
SUBTRACTOR_SYS_MESSAGE = "You will be given an integer, simply subtract one to it and return the result"
MULTIPLIER_SYS_MESSAGE = "You will be given an integer, simply multiply by 3 and return the result"
DIVIDER_SYS_MESSAGE = "You will be given an integer, simply DIVIDE by 3 and round down to the nearest integer, and return the result"
MASTER_SYS_MESSAGE = "You will be given a starting number and a target number. Use the other agents to turn the starting number into the target number. Then terminate group chat."
NUMBER_SYS_MESSAGE = "Simply return back the number."

# Descriptions
ADDER_DESCRIPTION = "Simply adds 1 to a given number."
SUBTRACTOR_DESCRIPTION = "Simply subtracts 1 to a given number."
MULTIPLIER_DESCRIPTION = "Simply multiples 3 to a given number."
DIVIDER_DESCRIPTION = "Simply divides 3 to a given number and rounds down to nearest integer."
NUMBER_DESCRIPTION = "Simply returns back the given number."

# Create agents
adder_agent = AssistantAgent(name="adder_agent", system_message=ADDER_SYS_MESSAGE, llm_config=llm_configuration, description=ADDER_DESCRIPTION)
subtractor_agent = AssistantAgent(name="subtractor_agent", system_message=SUBTRACTOR_SYS_MESSAGE, llm_config=llm_configuration, description=SUBTRACTOR_DESCRIPTION)
multiplier_agent = AssistantAgent(name="multiplier_agent", system_message=MULTIPLIER_SYS_MESSAGE, llm_config=llm_configuration, description=MULTIPLIER_DESCRIPTION)
divider_agent = AssistantAgent(name="divider_agent", system_message=DIVIDER_SYS_MESSAGE, llm_config=llm_configuration, description=DIVIDER_DESCRIPTION)
number_agent = AssistantAgent(name="number_agent", system_message=NUMBER_SYS_MESSAGE, llm_config=llm_configuration, description=NUMBER_DESCRIPTION)

# Register model client for each agent
for agent in [adder_agent, subtractor_agent, multiplier_agent, divider_agent, number_agent]:
    agent.register_model_client(model_client_cls=NeuroengineClient)

# Create group chat and group chat manager
group_chat = GroupChat(agents=[adder_agent, subtractor_agent, multiplier_agent, divider_agent, number_agent], messages=[])
group_chat_manager = GroupChatManager(name="master_agent", system_message=MASTER_SYS_MESSAGE, llm_config=llm_configuration, groupchat=group_chat)
group_chat_manager.register_model_client(model_client_cls=NeuroengineClient)

# Initiate chat
number_agent.initiate_chat(group_chat_manager, message="My number is 3, I want to turn it into 13.", summary_method="reflection_with_llm")

[autogen.oai.client: 07-25 11:32:55] {473} INFO - Detected custom model client in config: NeuroengineClient, model client can not be used until register_model_client is called.
[autogen.oai.client: 07-25 11:32:55] {473} INFO - Detected custom model client in config: NeuroengineClient, model client can not be used until register_model_client is called.
[autogen.oai.client: 07-25 11:32:55] {473} INFO - Detected custom model client in config: NeuroengineClient, model client can not be used until register_model_client is called.
[autogen.oai.client: 07-25 11:32:55] {473} INFO - Detected custom model client in config: NeuroengineClient, model client can not be used until register_model_client is called.
[autogen.oai.client: 07-25 11:32:55] {473} INFO - Detected custom model client in config: NeuroengineClient, model client can not be used until register_model_client is called.
[autogen.oai.client: 07-25 11:32:55] {473} INFO - Detected custom model client in config: NeuroengineClient, model client can not be used until register_model_client is called.
number_agent (to master_agent):

My number is 3, I want to turn it into 13.

--------------------------------------------------------------------------------
[autogen.oai.client: 07-25 11:32:55] {473} INFO - Detected custom model client in config: NeuroengineClient, model client can not be used until register_model_client is called.
Traceback (most recent call last):
  File "c:/Users/wirel/OneDrive/Documents/Python/neuro_test/main_groupchat.py", line 41, in <module>
    number_agent.initiate_chat(group_chat_manager, message="My number is 3, I want to turn it into 13.", summary_method="reflection_with_llm")
  File "C:\Users\wirel\OneDrive\Documents\Python\neuro_test\venv\lib\site-packages\autogen\agentchat\conversable_agent.py", line 1018, in initiate_chat
    self.send(msg2send, recipient, silent=silent)
  File "C:\Users\wirel\OneDrive\Documents\Python\neuro_test\venv\lib\site-packages\autogen\agentchat\conversable_agent.py", line 655, in send
    recipient.receive(message, self, request_reply, silent)
  File "C:\Users\wirel\OneDrive\Documents\Python\neuro_test\venv\lib\site-packages\autogen\agentchat\conversable_agent.py", line 818, in receive
    reply = self.generate_reply(messages=self.chat_messages[sender], sender=sender)
  File "C:\Users\wirel\OneDrive\Documents\Python\neuro_test\venv\lib\site-packages\autogen\agentchat\conversable_agent.py", line 1972, in generate_reply
    final, reply = reply_func(self, messages=messages, sender=sender, config=reply_func_tuple["config"])
  File "C:\Users\wirel\OneDrive\Documents\Python\neuro_test\venv\lib\site-packages\autogen\agentchat\groupchat.py", line 1047, in run_chat 
    speaker = groupchat.select_speaker(speaker, self)
  File "C:\Users\wirel\OneDrive\Documents\Python\neuro_test\venv\lib\site-packages\autogen\agentchat\groupchat.py", line 538, in select_speaker
    return self._auto_select_speaker(last_speaker, selector, messages, agents)
  File "C:\Users\wirel\OneDrive\Documents\Python\neuro_test\venv\lib\site-packages\autogen\agentchat\groupchat.py", line 658, in _auto_select_speaker
    result = checking_agent.initiate_chat(
  File "C:\Users\wirel\OneDrive\Documents\Python\neuro_test\venv\lib\site-packages\autogen\agentchat\conversable_agent.py", line 1011, in initiate_chat
    self.send(msg2send, recipient, request_reply=True, silent=silent)
  File "C:\Users\wirel\OneDrive\Documents\Python\neuro_test\venv\lib\site-packages\autogen\agentchat\conversable_agent.py", line 655, in send
    recipient.receive(message, self, request_reply, silent)
  File "C:\Users\wirel\OneDrive\Documents\Python\neuro_test\venv\lib\site-packages\autogen\agentchat\conversable_agent.py", line 818, in receive
    reply = self.generate_reply(messages=self.chat_messages[sender], sender=sender)
  File "C:\Users\wirel\OneDrive\Documents\Python\neuro_test\venv\lib\site-packages\autogen\agentchat\conversable_agent.py", line 1972, in generate_reply
    final, reply = reply_func(self, messages=messages, sender=sender, config=reply_func_tuple["config"])
  File "C:\Users\wirel\OneDrive\Documents\Python\neuro_test\venv\lib\site-packages\autogen\agentchat\conversable_agent.py", line 1340, in generate_oai_reply
    extracted_response = self._generate_oai_reply_from_client(
  File "C:\Users\wirel\OneDrive\Documents\Python\neuro_test\venv\lib\site-packages\autogen\agentchat\conversable_agent.py", line 1359, in _generate_oai_reply_from_client
    response = llm_client.create(
  File "C:\Users\wirel\OneDrive\Documents\Python\neuro_test\venv\lib\site-packages\autogen\oai\client.py", line 639, in create
    raise RuntimeError(
RuntimeError: Model client(s) ['NeuroengineClient'] are not activated. Please register the custom model clients using `register_model_client` or filter them out form the config list.

Hk669 commented 2 months ago

@wireless90 sorry for the issue, but we dont support NeuroengineClient, please do check out the blog on how to register a custom model client at https://microsoft.github.io/autogen/blog/2024/01/26/Custom-Models/

wireless90 commented 2 months ago

@wireless90 sorry for the issue, but we dont support NeuroengineClient, please do check out the blog on how to register a custom model client at https://microsoft.github.io/autogen/blog/2024/01/26/Custom-Models/

Hi, yes im using the custom model client. Agent to agent chat it works. Group chat its failing as mentioned above.

Downgrading works, as mentioned here.

https://github.com/microsoft/autogen/issues/2929#issuecomment-2242195049

# NeuroengineClient.py
from typing import Dict, Any, List
from neuroengine import Neuroengine
from openai.types.chat import ChatCompletion
from openai.types.completion_usage import CompletionUsage
from openai.types.chat.chat_completion import ChatCompletionMessage, Choice
from types import SimpleNamespace
import time

class NeuroengineClient:
    def __init__(self, config: Dict[str, Any], **kwargs):
        self.neuroengine = Neuroengine(
            service_name=config.get("service_name"),
            server_address=config.get("server_address"),
            server_port=config.get("server_port", 443),
            key=config.get("key", ""),
            verify_ssl=config.get("verify_ssl", True)
        )
        self.model_name = config["model"]
        self.id = 1

    def create(self, params: Dict[str, Any]) -> ChatCompletion:
        #print("Create method called with params:", params)  # Debug statement
        messages = params["messages"]

        # Convert the list of messages into a single prompt
        # TODO See if needs a prompt summary
        prompt = "\n".join([f'{msg["role"]}:{msg["content"]}' for msg in messages])

        response = self.neuroengine.request(prompt, max_new_len=4000)

        prompt_tokens = len(prompt.split())
        completion_tokens = len(response.split())
        total_tokens = prompt_tokens + completion_tokens

        message = ChatCompletionMessage(
            role="assistant",
            content=response,
            function_call=None,
            tool_calls=None
        )
        choices = [Choice(finish_reason="stop", index=0, message=message)]

        usage = SimpleNamespace(
            prompt_tokens=prompt_tokens,
            completion_tokens=completion_tokens,
            total_tokens=total_tokens
        )

        chat_completion = ChatCompletion(
            id=str(self.id),  # Generate or fetch a unique ID as per your requirements
            created=int(time.time()),
            object='chat.completion',
            model=self.model_name,
            choices=choices,
            #usage=usage,
            cost=0.0  # Adjust the cost calculation as needed
        )
        self.id += 1
        return chat_completion

    def message_retrieval(self, response: ChatCompletion) -> List[str]:
        return [choice.message.content for choice in response.choices]

    def cost(self, response: ChatCompletion) -> float:
        return 0.0

    @staticmethod
    def get_usage(response: ChatCompletion) -> Dict[str, Any]:
        return dict()

kaustubhrm commented 1 week ago

any fixes on this issue yet?

microsoft / autogen