🏆 🥇 Winner solution for Video Quality Assessment Challenge at the 1st AIS 2024 workshop @ CVPR 2024
Official Code for [CVPR Workshop 2024] Paper "COVER: A Comprehensive Video Quality Evaluator". Official Code, Demo, Weights for the Comprehensive Video Quality Evaluator (COVER).
This inspires us to develop a comprehensive and efficient model for the UGC VQA task
COVER contains three branches: (1) Semantic, (2) Aesthetic, and (3) Technical braches for comprehensive assessment of video quality. We use the semantic branch to gate the other two branches to conduct feature interactions. The final quality score is aggregated on top of the three scores.
Dataset: YT-UGC | SROCC | KROCC | PLCC | RMSE | Run Time* |
---|---|---|---|---|---|
TVQE (Wang et al, CVPRWS 2024) | 0.9150 | 0.7410 | 0.9182 | ------- | 705.30ms |
Q-Align (Zhang *et al, CVPRWS 2024) | 0.9080 | 0.7340 | 0.9120 | ------- | 1707.06ms |
SimpleVQA+ (Sun *et al, CVPRWS 2024) | 0.9060 | 0.7280 | 0.9110 | ------- | 245.51ms |
COVER | 0.9143 | 0.7413 | 0.9122 | 0.2519 | 🚀 79.37ms 🚀 |
The repository can be installed via the following commands:
git clone https://github.com/vztu/COVER
cd COVER
pip install -e .
mkdir pretrained_weights
cd pretrained_weights
wget https://github.com/vztu/COVER/blob/release/Model/COVER.pth
cd ..
You can run a single command to judge the quality of the demo videos in comparison with videos in VQA datasets.
python evaluate_one_image.py -i ./demo/image_1.mp4
or
python evaluate_one_video.py -v ./demo/video_2.mp4
Or choose any video you like to predict its quality:
python evaluate_one_video.py -v $YOUR_SPECIFIED_VIDEO_PATH$
The script can directly score the video's overall quality (considering all perspectives).
python evaluate_one_video.py -v $YOUR_SPECIFIED_VIDEO_PATH$
The final output score is the sum of all perspectives.
python evaluate_one_dataset.py -in $YOUR_SPECIFIED_DIR$ -out $OUTPUT_CSV_PATH$
python evaluate_a_set_of_videos.py -in $YOUR_SPECIFIED_DIR$ -out $OUTPUT_CSV_PATH$
The results are stored as .csv
files in cover_predictions in your OUTPUT_CSV_PATH
.
Please feel free to use COVER to pseudo-label your non-quality video datasets.
We have already converted the labels for most popular datasets you will need for Blind Video Quality Assessment, and the download links for the videos are as follows:
:book: LSVQ: Github
:book: KoNViD-1k: Official Site
:book: LIVE-VQC: Official Site
:book: YouTube-UGC: Official Site
(Please contact the original authors if the download links were unavailable.)
After downloading, kindly put them under the ../datasets
or anywhere but remember to change the data_prefix
respectively in the config file.
Now you can employ head-only/end-to-end transfer of COVER to get dataset-specific VQA prediction heads.
python transfer_learning.py -t $YOUR_SPECIFIED_DATASET_NAME$
For existing public datasets, type the following commands for respective ones:
python transfer_learning.py -t val-kv1k
for KoNViD-1k.python transfer_learning.py -t val-ytugc
for YouTube-UGC.python transfer_learning.py -t val-cvd2014
for CVD2014.python transfer_learning.py -t val-livevqc
for LIVE-VQC.As the backbone will not be updated here, the checkpoint saving process will only save the regression heads. To use it, simply replace the head weights with the official weights COVER.pth.
We also support end-to-end fine-tune right now (by modifying the num_epochs: 0
to num_epochs: 15
in ./cover.yml
). It will require more memory cost and more storage cost for the weights (with full parameters) saved, but will result in optimal accuracy.
You can be monitoring your results on WandB!
Thanks for every participant of the subjective studies!
Should you find our work interesting and would like to cite it, please feel free to add these in your references!
%AIS 2024 VQA challenge
@article{conde2024ais,
title={AIS 2024 challenge on video quality assessment of user-generated content: Methods and results},
author={Conde, Marcos V and Zadtootaghaj, Saman and Barman, Nabajeet and Timofte, Radu and He, Chenlong and Zheng, Qi and Zhu, Ruoxi and Tu, Zhengzhong and Wang, Haiqiang and Chen, Xiangguang and others},
journal={arXiv preprint arXiv:2404.16205},
year={2024}
}
%cover
@InProceedings{cover2024cpvrws,
author = {He, Chenlong and Zheng, Qi and Zhu, Ruoxi and Zeng, Xiaoyang and Fan, Yibo and Tu, Zhengzhong},
title = {COVER: A Comprehensive Video Quality Evaluator},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
month = {June},
year = {2024},
pages = {5799-5809}
}