Video-LLaVA: Learning United Visual Representation by Alignment Before Projection

If you like our project, please give us a star ⭐ on GitHub for latest update.

[](https://huggingface.co/spaces/LanguageBind/Video-LLaVA) [](https://openxlab.org.cn/apps/detail/jiaxicui/Video-LLaVA) [](https://modelscope.cn/studios/PKU-YuanLab/Video-LLaVA) [](https://replicate.com/nateraw/video-llava) [](https://arxiv.org/abs/2311.10122)
[](https://github.com/PKU-YuanGroup/Video-LLaVA/blob/main/LICENSE) [](https://hits.seeyoufarm.com) [](https://github.com/PKU-YuanGroup/Video-LLaVA/issues?q=is%3Aopen+is%3Aissue) [](https://github.com/PKU-YuanGroup/Video-LLaVA/issues?q=is%3Aissue+is%3Aclosed)
[](https://twitter.com/_nateraw/status/1726783481248977037) [](https://twitter.com/arankomatsuzaki/status/1726421417963516144) [](https://twitter.com/jesselaunz/status/1726850138776453379) [](https://mp.weixin.qq.com/s/EFqLv_Euf5VU024zOtzkkg) [](https://mp.weixin.qq.com/s/uwaxMu8UbJpcLTXsNJwpVQ) [](https://zhuanlan.zhihu.com/p/668166885) [](https://www.youtube.com/watch?v=EFkN00rGq1U&ab_channel=JesseLau-aTrader)

[](https://paperswithcode.com/sota/zeroshot-video-question-answer-on-msrvtt-qa?p=video-llava-learning-united-visual-1)
[](https://paperswithcode.com/sota/zeroshot-video-question-answer-on-msvd-qa?p=video-llava-learning-united-visual-1)
[](https://paperswithcode.com/sota/zeroshot-video-question-answer-on-tgif-qa?p=video-llava-learning-united-visual-1)

💡 I also have other video-language projects that may interest you ✨.

> [**Open-Sora-Plan**](https://github.com/PKU-YuanGroup/Open-Sora-Plan)
[](https://github.com/PKU-YuanGroup/Open-Sora-Plan) [](https://github.com/PKU-YuanGroup/Open-Sora-Plan)
> [**MoE-LLaVA: Mixture of Experts for Large Vision-Language Models**](https://arxiv.org/abs/2401.15947)
> Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, Li Yuan
[](https://github.com/PKU-YuanGroup/MoE-LLaVA) [](https://github.com/PKU-YuanGroup/MoE-LLaVA) [](https://arxiv.org/abs/2401.15947)
> [**LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment**](https://arxiv.org/abs/2310.01852)
> Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, Wancai Zhang, Zhifeng Li, Wei Liu, Li Yuan
[](https://github.com/PKU-YuanGroup/LanguageBind) [](https://github.com/PKU-YuanGroup/LanguageBind) [](https://arxiv.org/abs/2310.01852)

## 📰 News * **[2024.05.15]** 🤝🤝🤝 Thanks to the generous contributions of [@zucchini-nlp](https://github.com/zucchini-nlp), Video-LLaVa now available in the Transformers library! More details [here](https://github.com/PKU-YuanGroup/Video-LLaVA/issues/156). * **[2024.01.27]** 👀👀👀 Our [MoE-LLaVA](https://github.com/PKU-YuanGroup/MoE-LLaVA) is released! A sparse model with 3B parameters outperformed the dense model with 7B parameters. * **[2024.01.17]** 🔥🔥🔥 Our [LanguageBind](https://github.com/PKU-YuanGroup/LanguageBind) has been accepted at ICLR 2024! * **[2024.01.16]** 🔥🔥🔥 We reorganize the code and support LoRA fine-tuning, checking [finetune_lora.sh](scripts/v1_5/finetune_lora.sh). * **[2023.11.30]** 🤝 Thanks to the generous contributions of the community, the [OpenXLab's demo](https://openxlab.org.cn/apps/detail/houshaowei/Video-LLaVA) is now accessible. * **[2023.11.23]** We are training a new and powerful model. * **[2023.11.21]** 🤝 Check out the [replicate demo](https://replicate.com/nateraw/video-llava), created by [@nateraw](https://github.com/nateraw), who has generously supported our research! * **[2023.11.20]** 🤗 [Hugging Face demo](https://huggingface.co/spaces/LanguageBind/Video-LLaVA) and **all codes & datasets** are available now! Welcome to **watch** 👀 this repository for the latest updates. ## 😮 Highlights Video-LLaVA exhibits remarkable interactive capabilities between images and videos, despite the absence of image-video pairs in the dataset. ### 💡 Simple baseline, learning united visual representation by alignment before projection - With **the binding of unified visual representations to the language feature space**, we enable an LLM to perform visual reasoning capabilities on both images and videos simultaneously. ### 🔥 High performance, complementary learning with video and image - Extensive experiments demonstrate **the complementarity of modalities**, showcasing significant superiority when compared to models specifically designed for either images or videos. ## 🤗 Demo ### Gradio Web UI Highly recommend trying out our web demo by the following command, which incorporates all features currently supported by Video-LLaVA. We also provide [online demo](https://huggingface.co/spaces/LanguageBind/Video-LLaVA) in Huggingface Spaces. ```bash python -m videollava.serve.gradio_web_server ``` https://github.com/PKU-YuanGroup/Video-LLaVA/assets/62638829/71ab15ac-105e-4b18-b0b5-e1b35d70607b ### CLI Inference ```bash CUDA_VISIBLE_DEVICES=0 python -m videollava.serve.cli --model-path "LanguageBind/Video-LLaVA-7B" --file "path/to/your/video.mp4" --load-4bit ``` ```bash CUDA_VISIBLE_DEVICES=0 python -m videollava.serve.cli --model-path "LanguageBind/Video-LLaVA-7B" --file "path/to/your/image.jpg" --load-4bit ``` ## 🚀 Main Results ### Image understanding

### Video understanding

## 🛠️ Requirements and Installation * Python >= 3.10 * Pytorch == 2.0.1 * CUDA Version >= 11.7 * Install required packages: ```bash git clone https://github.com/PKU-YuanGroup/Video-LLaVA cd Video-LLaVA conda create -n videollava python=3.10 -y conda activate videollava pip install --upgrade pip # enable PEP 660 support pip install -e . pip install -e ".[train]" pip install flash-attn --no-build-isolation pip install decord opencv-python git+https://github.com/facebookresearch/pytorchvideo.git@28fe037d212663c6a24f373b94cc5d478c8c1a1d ``` ## 🤖 API > [!Warning] > ``` pip install -U transformers ``` If you need to install `av` then do ``` python -m pip install av ``` ``` import av import numpy as np from transformers import VideoLlavaProcessor, VideoLlavaForConditionalGeneration def read_video_pyav(container, indices): frames = [] container.seek(0) start_index = indices[0] end_index = indices[-1] for i, frame in enumerate(container.decode(video=0)): if i > end_index: break if i >= start_index and i in indices: frames.append(frame) return np.stack([x.to_ndarray(format="rgb24") for x in frames]) model = VideoLlavaForConditionalGeneration.from_pretrained("LanguageBind/Video-LLaVA-7B-hf") processor = VideoLlavaProcessor.from_pretrained("LanguageBind/Video-LLaVA-7B-hf") prompt = "USER: