ConvLLaVA: Hierarchical Backbones as Visual Encoder for Large Multimodal Models
[Chunjiang Ge](https://john-ge.github.io/), [Sijie Cheng](https://adacheng.github.io/), Ziming Wang, Jiale Yuan, Yuan Gao Jun Song, Shiji Song, [Gao Huang](https://www.gaohuang.net/), Bo Zheng
[ English | 中文 ]
Abstract
High-resolution Large Multimodel Models (LMM) encounter the challenges of excessive visual tokens and quadratic visual complexity. Current high-resolution LMMs address the quadratic complexity while still generating excessive visual tokens. However, the redundancy in visual tokens is the key problem as it leads to more substantial compute. To mitigate this, we propose ConvLLaVA, which employs ConvNeXt, a hierarchical backbone, as the visual encoder of LMM to replace Vision Transformer (ViT). ConvLLaVA compresses high-resolution images into information-rich visual features, effectively avoiding the generation of excessive visual tokens. To enhance the capabilities of ConvLLaVA, we propose two critical optimizations.
- Since the low-resolution pretrained ConvNeXt underperforms when directly applied on high resolution, we update it to merge the gap.
- Furthermore, since ConvNeXt's original compression ratio is insufficient for much higher resolution inputs, we train a successive stage to further compress the visual tokens, effectively reducing redundancy.
These optimizations enable ConvLLaVA to support inputs of 1536x1536 resolution while generating only 576 visual tokens, accommodating images of arbitrary aspect ratios. Experimental results demonstrate that our method achieves competitive performance with state-of-the-art models on mainstream benchmarks.
Release :loudspeaker:
- 2024/05/25: Checkpoints are released.
- 2024/04/17: Our code is released.
If you are interested in Large Multimodal Models or you have great ideas, please feel free to email with me: Chunjiang Ge.
Usage and License Notices: This project utilizes certain datasets and checkpoints that are subject to their respective original licenses. Users must comply with all terms and conditions of these original licenses, including but not limited to the OpenAI Terms of Use for the dataset and the specific licenses for base language models for checkpoints trained using the dataset (e.g. Llama community license for LLaMA-2 and Vicuna-v1.5). This project does not impose any additional constraints beyond those stipulated in the original licenses. Furthermore, users are reminded to ensure that their use of the dataset and checkpoints is in compliance with all applicable laws and regulations.
Contents
- Abstract
- Release :loudspeaker:
- Contents
- TODO
- Install
- Model Zoo
- Dataset
- Train
- Evaluation
- Citation
- Acknowledgement
TODO
- [ ] Add LMMs-eval supports.
- [ ] Add VLMEvalKit supports.
- [ ] Add xtuner supports.
- [x] Release weights.
- [ ] Release inference code.
Install
-
Clone this repository and navigate to ConvLLaVA folder
git clone https://github.com/alibaba/conv-llava cd conv-llava -
Install Package
conda create -n convllava python=3.11 -y conda activate convllava pip install --upgrade pip # enable PEP 660 support pip install -e . -
Install additional packages for training cases
pip install -e ".[train]" pip install flash-attn --no-build-isolation
Model Zoo
The performance on mainstream benchmarks are shown below:
| Method | Resolution | Visual Tokens | LLM | MME | MMB | SEED | RealWorldQA | MMMU | MMVet | Text | Doc | POPE |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ConvLLaVA | 768 | 144 | 7B | 1541 | 68 | 68.8 | 55.9 | 36.3 | 44.8 | 59.1 | 44.8 | 87.3 |
| ConvLLaVA | 1024 | 256 | 7B | 1553 | 68.8 | 69.3 | 58.8 | 35.1 | 44.4 | 62.5 | 48.5 | 87.7 |
| ConvLLaVA | 1536 | 576 | 7B | 1575 | 68.7 | 70.2 | 59.9 | 35.8 | 45.9 | 65.8 | 59 | 87.3 |
| Method | Resolution | Visual Tokens | LLM | RefCOCO | RefCOCO+ | RefCOCOg | Avg | |||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| val | test-A | test-B | val | test-A | test-B | val | test | |||||
| ConvLLaVA | 768 | 144 | 7B | 84.5 | 89.0 | 79.2 | 77.7 | 84.9 | 69.7 | 79.8 | 79.7 | 80.6 |
| ConvLLaVA | 1024 | 256 | 7B | 85.5 | 89.6 | 78.8 | 79.3 | 86.1 | 70.3 | 80.6 | 81.2 | 81.4 |
| ConvLLaVA | 1536 | 576 | 7B | 86.5 | 90.6 | 80.5 | 80.0 | 86.8 | 71.5 | 82.0 | 82.4 | 82.3 |
Please check out our Model Zoo for all public ConvLLaVA checkpoints, and the instructions of how to use the weights.
Dataset
Data we use is introduded in Data.md.
Train
We use the following hyperparameters for training ConvLLaVA.
| Hyperparameters | Stage 1 | Stage 2 | Stage 3 |
|---|---|---|---|
| Learning Rate | 3e-4 | 2e-5 | 2e-5 |
| Batch Size | 256 | 256 | 128 |
| Epochs | 1 | 1 | 1 |
| Warmup Ratio | 0.03 | 0.03 | 0.03 |
| Weight Decay | 0 | 0 | 0 |
| Optimizer | AdamW | AdamW | AdamW |
The training scripts are in the scripts:
Evaluation
We support VLMEVALKIT and lmms-eval to evaluate our model now. See Evaluation.md for more details.
Citation
If you find LLaVA useful for your research and applications, please cite using this BibTeX:
@misc{ge2024convllava,
title={ConvLLaVA: Hierarchical Backbones as Visual
Encoder for Large Multimodal Models},
author={Chunjiang Ge, Sijie Cheng, Ziming Wang, Jiale Yuan, Yuan Gao, Jun Song, Shiji Song, Gao Huang, Bo Zheng},
archivePrefix={arXiv},
primaryClass={cs.CV}
year={2024}
eprint={2045.15738},
}