search

fudan-generative-vision / champ

Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidance
https://fudan-generative-vision.github.io/champ/
MIT License
4.33k stars 537 forks source link

the code of computing PSNR in the Disco repository is wrong #8

Closed Delicious-Bitter-Melon closed 5 months ago

Delicious-Bitter-Melon commented 6 months ago

The Disco code of computing PSNR as follows is wrong: https://github.com/Wangt-CN/DisCo/blob/8538889c9ee9edd8dd43ffee182d1a91ce7a9828/tool/metrics/ssim_l1_lpips_psnr.py#L13.

image

As pointed out in https://github.com/Wangt-CN/DisCo/issues/86, the accurate code is mse = np.mean((original/1.0 - compressed/1.0) 2) instead of mse = np.mean((original - compressed) 2) , because original and compressed images are uint8 in their code, and (original - compressed) 2 will cause numerical overflow.

If you use their evaluation code of computing PSNR, please update your results.

Delicious-Bitter-Melon commented 6 months ago

The wrong version will improve PSNR. For example, image

ShenhaoZhu commented 6 months ago

Thank you for the reminder. To ensure a fair comparison, we follow the method for calculating PSNR that was used by Disco and subsequent works.

Delicious-Bitter-Melon commented 6 months ago

Thank you for the reminder. To ensure a fair comparison, we follow the method for calculating PSNR that was used by Disco and subsequent works.

In some cases, the wrong computation manner will increase PSNR but in some cases it decreases PSNR. Are you sure that your conclusion in your paper will not change after using the correct PSNR calculation method?

image

Delicious-Bitter-Melon commented 6 months ago

Besides, I am certain that the FID calculation method used by Disco is correct, but I am not sure if other metrics such as FVD are correct.