[Bug]: speaker_selection_agent does not get registered in GroupChat if the selector is a custom model client

[Basekill]

Describe the bug

When creating a group chat with speaker_selection_method="auto" and the group chat manager registered as a custom model client, the speaker_selection_agent gets the custom llm_config, but is not registered with the custom model client. Thus we get a model client not activated error when trying to use the speaker_selection_agent at https://github.com/microsoft/autogen/blob/b5d856dd36abfda603c39b36979da2d0636e00f5/autogen/agentchat/conversable_agent.py#L1344

Here the speaker_selection_agent is not registered: https://github.com/microsoft/autogen/blob/b5d856dd36abfda603c39b36979da2d0636e00f5/autogen/agentchat/groupchat.py#L625

This also occurs in other places where an agent is created for you. For example, in Teachability, self.analyzer is created for you without registration of custom model.

Steps to reproduce

1. Create a GroupChat with speaker_selection_method="auto"
2. Create a GroupChatManager with a custom model client as llm_config
3. Register the GroupChatManager with the custom model client
4. Initiate a chat with the GroupChatManager
5. See model client not activated error

Model Used

Custom model

Expected Behavior

speaker_selection_agent should be registered with the same model client as the selector

Screenshots and logs

Custom LLM client does not get registered here

Additional Information

No response

[ekzhu]

Thanks for pointing this out. What do you think about exposing the select_speaker_agent either through constructor or an attribute, so we can better customize the llm_config and its client.

cc @WaelKarkoub @marklysze

[Basekill]

Although exposing select_speaker_agent would solve this problem in this case, it is a general issue with agents being initialised for you in methods (internal helper agents that are not created by the developer but in the library).

For example, the self.analyzer created for Teachability (https://github.com/microsoft/autogen/blob/d50f6547487e57edf5b805ccc32512a31afa571c/autogen/agentchat/contrib/capabilities/teachability.py#L77)

Is not registered to a model client for custom models.

I'm sure there are other places where an agent is used internally but is not registered.

One solution could be to store a list of registered custom model client classes, then when an internal agent is used, it would register all the model client classes in that list.

If we were to expose the internal agents, this would have to be done for all internal agents - which might be revealing too much about internal implementation detail if the implementation were to be changed in the future. Although it would add more customizability as it means the internal agents could have a different client to the outer agent so it is a trade-off.

[ekzhu]

One solution could be to store a list of registered custom model client classes, then when an internal agent is used, it would register all the model client classes in that list.

This is a good idea. We can start by simply passing the client of the parent agent to the internal agent, starting from the group chat.

[Matteo-Frattaroli]

Same problem here.

I agree with @Basekill 's statement:

it is a general issue with agents being initialised for you in methods (internal helper agents that are not created by the developer but in the library).

@ekzhu maybe some kind of custom_model detection strategy could be defined, perhaps detecting the presence of 'model_client_cls' in the custom model's llm_config (even though I'm not sure if this is used by oai/azure oai models too). In this way it would be possible to detect a custom model that needs registration and call register_model_client on any agent instance created within the library.

If 'model_client_cls' is already used by oai/azure oai perhaps a new key that distinguishes custom models could be required in the config.

[sonichi]

There is one approach to solve the problem in a generic manner: Move the register_model_client out of the agents. Instead, make it a function to manipulate configs. Then, pass the materialized configs to agents so that the agents never need to worry about register_model_client.