search

Tencent / DVQA

Deep learning-based Video Quality Assessment
471 stars 89 forks source link

a question about our eval results #5

Closed harlanc closed 4 years ago

harlanc commented 4 years ago

Hi tommyhq,

We now use your model to evaluate our own video samples,

image

the only differences between the samples are video bitrate, the bitrate of reference video is 1600Kbit/s, and the 3 dis ones are 400kbit/s ,800kbit/s and 1200kbits/s,

Why higher bitrates has a lower PRED score??

image
tommyhq commented 4 years ago

Thanks for pointing out this issue.

First, I noticed the bitrate of reference video is 1600 Kbit/s, and I believe this is not the typical, i.e. lossless, 'reference' for a FR-VQA metric.

Second, the framerate is 16 fps. That is a number rarely encountered.

Could you give more information about the sequence? For example, how the sequence was captured? Any preprocessing on the most original reference to get the 1600Kbit/s 'reference' ? Meanwhile, is it possible to upload those sequences so we could have a look?

harlanc commented 4 years ago

I didn't read README carefully:

We are not sure about the performance when the code is run with the following scenario,

PGC with other distortion types, especially time-related distortions. PGC with post-processing filters, like de-nosing, super-resolution, artifacts reduction, etc. UGC videos with pre-processing filter. UGC videos compressed with common codecs.

We tried to eval a UGC dataset using the pretrained model, this usage was wrong since the model was trained on a PGC dataset.

Next maybe we will train our own model on UGC dataset. Thanks for @tommyhq's help.