search

universome / stylegan-v

[CVPR 2022] StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2
https://universome.github.io/stylegan-v
333 stars 36 forks source link

The frame sequence is out of order when evaluating fvd #43

Open RenShuhuai-Andy opened 2 months ago

RenShuhuai-Andy commented 2 months ago

Hi @universome, thanks for your great work!

I followed your instructions to evaluate fvd2048_16f based on calc_metrics_for_dataset.py, however, I found a potential bug.

Specifically, I prepared the ground truth (gt) and predicted datasets according to the following dataset structure:

dataset/
    video1/
        - frame1.jpg
        - frame2.jpg
        - ...
    video2/
        - frame1.jpg
        - frame2.jpg
        - ...
    ...

However, the code in https://github.com/universome/stylegan-v/blob/master/src/training/dataset.py#L319 will sort the frames in lexicographic order of file names (frame1, frame10, frame11, frame12...), instead of numerical order (frame1, frame2, frame3, frame4...): image

I wonder if this out-of-order frame sequences lead to inaccurate evaluation results because I3D features are out of order?