[Feature Request] Integrate open source multi-modal models

[Wendong-Fan]

Required prerequisites

• [X] I have searched the Issue Tracker and Discussions that this hasn't already been reported. (+1 or comment there if it has.)
• [x] Consider asking first in a Discussion.

Motivation

before the implementation, study on what's great open source multi-modal models in the market and check whether it has been supported by ollama or litellm, if not supported, then we can do native integration

Solution

https://x.com/wjqdev/status/1808751706970489287?s=46

1. open-source model
2. image, text input
3. function/tool calling

Alternatives

No response

Additional context

No response

[Wendong-Fan]

also take a look at open source models support function calling

[Wendong-Fan]

example: llava series