Fail to create custom dataset statistics for KID computation

[BrandoZhang]

Can I create custom dataset statistics to compute KID?

In my trial, after make_custom_stats, the invoking of compute_fid works fine but compute_kid tries to download the statistics:

>>> fid.compute_kid('sunglasses/0', dataset_name='sunglasses', mode='clean', dataset_res=256, dataset_split='custom')
compute KID of a folder with sunglasses statistics
downloading statistics to /home/brando/.local/lib/python3.7/site-packages/cleanfid/stats/sunglasses_clean_custom_na_kid.npz
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/brando/.local/lib/python3.7/site-packages/cleanfid/fid.py", line 335, in compute_kid
    mode=mode, seed=0, split=dataset_split, metric="KID")
  File "/home/brando/.local/lib/python3.7/site-packages/cleanfid/features.py", line 95, in get_reference_statistics
    fpath = check_download_url(local_folder=stats_folder, url=url)
  File "/home/brando/.local/lib/python3.7/site-packages/cleanfid/downloads_helper.py", line 27, in check_download_url
    with urllib.request.urlopen(url) as response, open(local_path, 'wb') as f:
  File "/usr/lib/python3.7/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.7/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.7/urllib/request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.7/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.7/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

Detailed context can be found in https://github.com/GaParmar/clean-fid/issues/9.

[Baran-phys]

I have the same problem for FID and get the same error.

[GaParmar]

Can you show the error message that you get?

[Baran-phys]

I also have a question. Shouldn't one normalize (Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]) the input data to the inceptionv3W model?

[GaParmar]

For the custom stats error, it looks like your custom dataset statistics file was not created. Can you check the files in the folder os.path.join(os.path.dirname(cleanfid.__file__), "stats")

[GaParmar]

Looking at this lines you do not need these normalization. jit detector If you look at the forward pass function defined within the torchscript inception model, the normalization applied is the same is the one in cleanfid's inceptionv3W class.

[Baran-phys]

Hmm, interesting. Is this such normalization crucial due to the fact that the InceptionV3 takes inputs as ((x-128)/128) and had been trained with these normalized inputs? I am asking since I retained timm https://rwightman.github.io/pytorch-image-models/models/inception-v3/ on a totally different and custom dataset without this normalization and I wonder if I have to stick to no-normalization still or I have to adhere to a certain regime of normalization for the retrained inceptionv3 no matter what.