LMM: use `_generate_oai_reply_from_client`

BabyCNM commented 1 month ago

Why are these changes needed?

This PR only has one-line change (as major change) and a new test case. However, it addresses several issues mentioned before: https://github.com/microsoft/autogen/issues/2550 https://github.com/microsoft/autogen/issues/3507

Details:

In 2024, we introduced the function _generate_oai_reply_from_client in Conversable Agent (https://github.com/microsoft/autogen/pull/1575), which can handle function calling, model dumping, reflection, and many other great features. However, this change is not updated in the MultimodalConversableAgent, which causes lots of issues afterwards for the multimodal agent in function calling, group chat, and many other locations.

The fix is simple.

Updates:

In MultimodalConversableAgent, override the OAI's client call with _generate_oai_reply_from_client.
We also added a new test cases accordingly.
We now use gpt-4-turbo as the default model instead of gpt-4-vision-preview, as the preview model is deprecated by OpenAI now.

Related issue number

Checks

[X] I've included any doc changes needed for https://autogen-ai.github.io/autogen/. See https://autogen-ai.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
[X] I've added tests (if relevant) corresponding to the changes introduced in this PR.
[X] I've made sure all auto checks have passed.

BabyCNM commented 1 month ago

NOTE: the test case is failing because of API key. Let me know how to address it.

openai.AuthenticationError: Error code: 401 - {'error': {'message': 'Incorrect API key provided: sk-mocko***************************************only. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}

Hk669 commented 1 month ago

NOTE: the test case is failing because of API key. Let me know how to address it.

openai.AuthenticationError: Error code: 401 - {'error': {'message': 'Incorrect API key provided: sk-mocko***************************************only. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}

We can just skip the tests for this, If it is tested locally on all OS.

Hk669 commented 1 month ago

@BabyCNM im on windows, let me know if it is already tested.

BabyCNM commented 1 month ago

@Hk669 Thanks! I have tested in Mac OS.

qingyun-wu commented 1 month ago

@BabyCNM im on windows, let me know if it is already tested.

@Hk669 did you test it on windows? Thank you!

Hk669 commented 1 month ago

@Hk669 did you test it on windows? Thank you!

I couldn't test it, let me do it tomorrow. But it looks good to me though.

sonichi commented 1 month ago

The LMM tests should be skiped when skip-openai is specified. Please add a skip condition similar to other tests in contrib tests, and add the test back to contrib-openai CI. @qingyun-wu I think the LMM tests were removed from contrib-openai CI before. Could you chime in if there's an issue that prevents this test to be added to CI?

qingyun-wu commented 1 month ago

The LMM tests should be skiped when skip-openai is specified. Please add a skip condition similar to other tests in contrib tests, and add the test back to contrib-openai CI. @qingyun-wu I think the LMM tests were removed from contrib-openai CI before. Could you chime in if there's an issue that prevents this test to be added to CI?

Ok. Will take a look at this and get back soon.

autogenhub / autogen