search

RenShuhuai-Andy / TimeChat

[CVPR 2024] TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding
https://arxiv.org/abs/2312.02051
BSD 3-Clause "New" or "Revised" License
267 stars 23 forks source link

how to evaluation on activitynet-DVC? #6

Closed TXH-mercury closed 7 months ago

TXH-mercury commented 8 months ago

Thanks for the good job! I want to evaluate dvc on activitynet captions, there are val1 and val2 two annotations, how to modify the code?

RenShuhuai-Andy commented 8 months ago

Hi, following previous work (e.g., https://github.com/jayleicn/recurrent-transformer), we can use val1 as a val set, and val2 as a test set.

To facilitate evaluation, we provide the processed coco-format annotations in https://huggingface.co/datasets/ShuhuaiRen/TimeIT/tree/main/data/dense_video_captioning/anet. The script for evaluating anet dense captioning can be found in https://github.com/RenShuhuai-Andy/TimeChat/blob/master/eval.sh#L15-L21.

If you want to process other custom evaluation data, please refer to https://github.com/RenShuhuai-Andy/TimeChat/blob/master/docs/DATA.md#process-custom-evaluation-data-into-coco-format.

RenShuhuai-Andy commented 8 months ago

Hi, the annotation files and processing code that I uploaded yesterday have something wrong. I have updated them just now. Please re-download them, sorry about that.

We recommend to run our processing code by yourself to ensure that all the broken local videos are filtered.

RenShuhuai-Andy commented 8 months ago

For your information, we evaluate the performance of TimeChat-7B on anet dvc test set (val2), the results are: image