The following checkpoints store learnable parameters (positional embedding layers, Time-aware Frame Encoder, Sliding Video Q-Former, linear projection layers, and lora) only.
Checkpoint | LLM backbone | Link | Note |
---|---|---|---|
TimeChat-2-7B-Finetuned | LLaMA-2 7B | link | Fine-tuned on the instruction-tuning data from TimeIT-104K (asr version) and Valley-73K (previous version of current Valley-65K) |
First, install ffmpeg.
apt update
apt install ffmpeg
Then, create a conda environment:
conda env create -f environment.yml
conda activate timechat
pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu113
Before fine-tuning your own model (or reproduce our TimeChat model), make sure you have obtained the following checkpoints:
wget https://storage.googleapis.com/sfr-vision-language-research/LAVIS/models/BLIP2/eva_vit_g.pth
wget https://storage.googleapis.com/sfr-vision-language-research/LAVIS/models/InstructBLIP/instruct_blip_vicuna7b_trimmed.pth
Use git-lfs
to download weights of Video-LLaMA (7B):
git lfs install
git clone https://huggingface.co/DAMO-NLP-SG/Video-LLaMA-2-7B-Finetuned
git lfs install
git clone https://huggingface.co/ShuhuaiRen/TimeChat-7b
The file structure looks like:
ckpt/
|ββ Video-LLaMA-2-7B-Finetuned/
|-- llama-2-7b-chat-hf/
|-- VL_LLaMA_2_7B_Finetuned.pth
|ββ instruct-blip/
|-- instruct_blip_vicuna7b_trimmed.pth
|ββ eva-vit-g/
|-- eva_vit_g.pth
|-- timechat/
|-- timechat_7b.pth
Please refer to our Jupyter Demo here.
For now, the fine-tuning dataset consists of:
Config the checkpoint and dataset paths in stage2_finetune_time104k_valley72k.yaml.
conda activate timechat
torchrun --nproc_per_node=8 train.py --cfg-path train_configs/stage2_finetune_time104k_valley72k.yaml
Config the checkpoint and dataset paths in timechat.yaml.
Config the downstream task in eval.sh.
bash eval.sh
We are grateful for the following awesome projects our TimeChat arising from:
Our TimeChat is just a research preview intended for non-commercial use only. You must NOT use our TimeChat for any illegal, harmful, violent, racist, or sexual purposes. You are strictly prohibited from engaging in any activity that will potentially violate these guidelines.
If you find our project useful, hope you can star our repo and cite our paper as follows:
@article{Ren2023TimeChat,
title={TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding},
author={Shuhuai Ren and Linli Yao and Shicheng Li and Xu Sun and Lu Hou},
journal={ArXiv},
year={2023},
volume={abs/2312.02051},
}