WeMM

[📖中文版本] [🤗 HuggingFace]

WeMM is the most recent version of the multimodal large language model introduced by WeChatCV. Capable of engaging in Chinese-English bilingual conversations alongside dynamic high-resolution images, WeMM stands as the most powerful model with 10 billion parameters in the multimodal large language model rankings. Its overall evaluation score (Avg Rank) places it within the top tier. (Refer to vlm_leaderboard)

News 🚀🚀🚀

• 2024/06/27: 🚀🚀🚀🔥🔥🔥 The WeMM-Chat-2K-CN model is now available, supporting 2K resolution image input.
• 2024/06/21: 🚀🚀🚀 The WeMM-Chat-CN model has been launched, improving the model's bilingual dialogue capabilities in both Chinese and English.
• 2024/06/09: 🚀🚀🔥 WeMM-1.2(VL) model weight and test code are released, ranking first among 10-billion-parameter models in all MLLM benchmark tests.
• 2023/11/20: Model weight, inference code, and MME evaluation code have been released.
• 2023/11/13: 🚀 WeMM-1.0 scored 2066.66 points in the full set of MME benchmark tests, ranking first at the time. Its perception score was 1621.66, ranking first, and its cognition score was 445, ranking third.
• 2023/10/13: WeMM-1.0 achieved a perception score of 1551.63 on the MME benchmark, representing the state-of-the-art performance at that time.
• 2023/09/13: WeMM scored 68.6 points on the MMBench leaderboard, ranking second place at the time.

Model Zoo

Multimodal

WeMM is a multimodal language model with bilingual dialogue capabilities in Chinese and English. The evaluation results for its related classic tasks are as follows.

In all benchmark evaluations, WeMM uses the comprehensive average ranking (Avg Rank) as the primary performance reference indicator due to the significant differences in score distribution among different benchmarks. WeMM ranks first among 10-billion-parameter models and is also in the top tier among all MLLM models.

Parameters < 20B	All VLLM

Requirements

• transformers<=4.40.0 (and >=4.38.0)
• python >= 3.8
• torch >= 2.0.0
• flash-attn
• numpy==1.23.5
• sentencepiece==0.1.99
• ninja==1.11.1
• wheel
• cos-python-sdk-v5
• peft
• einops

For detailed requirements, please refer to requirements.txt.

Inference

Feel free to visit our github for more execution code.

The following code will initiate an interactive interface that allows users to input an image path and prompt to receive a response from WeMM.

• WeMM-1.2（VL）

  
  import torch
  from PIL import Image
  from transformers import AutoModel, GenerationConfig

model_path = 'feipengma/WeMM' # the path to the model wemm = AutoModel.from_pretrained(model_path, torch_dtype=torch.bfloat16, trust_remote_code=True) wemm.cuda() wemm.eval()

query = 'Describe this image.' image = 'path/to/your/image' pred = wemm.mm_generate(image, query)

+ WeMM-Chat-CN
```python
import torch
from PIL import Image
from transformers import AutoModel, GenerationConfig

model_path = 'feipengma/WeMM-Chat-CN'
wemm = AutoModel.from_pretrained(model_path, torch_dtype=torch.bfloat16, trust_remote_code=True)
wemm.cuda()
wemm.eval()

query = 'Why is the ocean blue?'
image = 'path/to/your/image'
pred = wemm.mm_generate(image, query)

• WeMM-Chat-2K-CN

  
  import torch
  from PIL import Image
  from transformers import AutoModel, GenerationConfig

model_path = 'feipengma/WeMM-Chat-2k-CN' wemm = AutoModel.from_pretrained(model_path, torch_dtype=torch.bfloat16, trust_remote_code=True) wemm.cuda() wemm.eval()

query = 'Why is the ocean blue?' image = 'path/to/your/image' pred = wemm.mm_generate(image, query)

## Example
**查看中文示例在[\[📖中文版本\]](./README_cn.md)**
+ Example 1

|Image| Conversation Example |
|:--:|:---:|
|  <img src="https://github.com/scenarios/WeMM/raw/main/assets/sample_1_fig.jpg" width="300">   |  <img src="https://github.com/scenarios/WeMM/raw/main/assets/sample_1_en.png" width="600">   |

+ Example 2

|Image| Conversation Example |
|:--:|:---:|
|  <img src="https://github.com/scenarios/WeMM/raw/main/assets/sample_2_fig.png" width="400">   |  <img src="https://github.com/scenarios/WeMM/raw/main/assets/sample_2_en.png" width="600">   |

+ Example 3

|Image | Conversation Example |
|:--:|:---:|
|  <img src="https://github.com/scenarios/WeMM/raw/main/assets/sample_4_fig.jpg" width="300">   |  <img src="https://github.com/scenarios/WeMM/raw/main/assets/sample_4_en.png" width="600">   |

## Acknowledgement
In our WeMM research, we referred to the base vision backbone code incorporating the navit980 structure from [Idefics2](https://huggingface.co/HuggingFaceM4/idefics2-8b), as well as the LLM framework from [Internlm2](https://huggingface.co/internlm/internlm2-chat-7b). We greatly appreciate the outstanding work of Idefics2 and InternLM2.

## License

This project is released under the [MIT license](LICENSE). Parts of this project contain code and models from other sources, which are subject to their respective licenses.

## Main Participating Institutions
+ Tencent WeChatCV Team
+ Zhejiang University