LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models

Yuzhang Shang*, Mu Cai*, Bingxin Xu, Yong Jae Lee^, Yan Yan^

*Equal Contribution, ^Equal Advising

[Paper] [Project Page]

How to run

Step.0: Set the environment the same as LLaVA-1.5

Note that the core of our proposed module is here in the CLIP image encoder.

Step.1 (for inference): Download Checkpoints

Download the checkpoints (LoRA Version) from Yuzhang's Huggingface Homepage to checkpoints/llava-v1.5-7b-lora-prunemerge.

Step.2 (for inference): Change the methods (PruMerge or PruMerge+).

Change the call function of token reduction from here in the CLIP image encoder.

Step.3 (for inference): Run the script.

For example, the evaluation for TextVQA is:

CUDA_VISIBLE_DEVICES=7 XDG_CACHE_HOME='/data/shangyuzhang/' bash scripts/v1_5/eval/testvqa.sh

For other inference scripts, refer to LLaVA Evaluation.