A library that contains a rich collection of performant PyTorch model metrics, a simple interface to create new metrics, a toolkit to facilitate metric computation in distributed training and tools for PyTorch model evaluations.
The model and weight of InceptionV3 is not strictly same to the TensorFlow version, which is used by most papers. Now I replace it with InceptionV3 model from pytorch-fid.
The insufficient precision of torcheval.metrics.image.fid.FrechetInceptionDistance._calculate_frechet_distance leads to wrong results. I fix this by setting all variables (fake_sum, real_cov_sum, ...) to float64.
With these modifications, the FID results are aligned. I suggest the maintainers reimplement the FID to align it with other libraries, otherwise researchers won't use torcheval to calculate FID.
Versions
PyTorch version: 2.0.1+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A
🐛 Describe the bug
Just use FrechetInceptionDistance on any images.
I have found the causes.
torcheval.metrics.image.fid.FrechetInceptionDistance._calculate_frechet_distance
leads to wrong results. I fix this by setting all variables (fake_sum
,real_cov_sum
, ...) to float64.With these modifications, the FID results are aligned. I suggest the maintainers reimplement the FID to align it with other libraries, otherwise researchers won't use torcheval to calculate FID.
Versions
PyTorch version: 2.0.1+cu117 Is debug build: False CUDA used to build PyTorch: 11.7 ROCM used to build PyTorch: N/A
OS: Ubuntu 18.04.5 LTS (x86_64) GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 Versions of relevant libraries: [pip3] flake8==7.0.0 [pip3] mypy-extensions==1.0.0 [pip3] numpy==1.24.4 [pip3] pytorch-fid==0.3.0 [pip3] pytorch-lightning==1.4.2 [pip3] torch==2.0.1 [pip3] torch-ema==0.3 [pip3] torch-fidelity==0.3.0 [pip3] torchaudio==2.0.2 [pip3] torcheval==0.0.7 [pip3] torchmetrics==0.5.0 [pip3] torchvision==0.15.2 [pip3] triton==2.0.0 [conda] blas 1.0 mkl
[conda] cudatoolkit 11.3.1 h2bc3f7f_2
[conda] ffmpeg 4.3 hf484d3e_0 pytorch [conda] mkl 2021.4.0 h06a4308_640
[conda] mkl-service 2.4.0 py38h7f8727e_0
[conda] mkl_fft 1.3.1 py38hd3c417c_0
[conda] mkl_random 1.2.2 py38h51133e4_0
[conda] numpy 1.24.4 pypi_0 pypi [conda] pytorch-fid 0.3.0 pypi_0 pypi [conda] pytorch-lightning 1.4.2 pypi_0 pypi [conda] pytorch-mutex 1.0 cuda pytorch [conda] torch 2.0.1 pypi_0 pypi [conda] torch-ema 0.3 pypi_0 pypi [conda] torch-fidelity 0.3.0 pypi_0 pypi [conda] torchaudio 2.0.2 pypi_0 pypi [conda] torcheval 0.0.7 pypi_0 pypi [conda] torchmetrics 0.5.0 pypi_0 pypi [conda] torchvision 0.15.2 pypi_0 pypi [conda] triton 2.0.0 pypi_0 pypi