Regarding FID scores

[sakethbachu]

Hi,

I obtained the FID score using the compute-fid.sh file but the score turns out to be 55.54 (I ran the test.sh script with all the same parameters as mentioned in the repo)

How can I improve this?

Thanks in advance!

[guillaumejs2403]

Hi @sakethbachu,

That is weird. Did you look at the explanations? Did you check flip rate? It should be around ~97%. Also, make sure that you are running DiME in the complete set: i.e. use the flag --dataset CelebA and not --dataset CelebAMW.

Best,

[sakethbachu]

Hi @sakethbachu,

That is weird. Did you look at the explanations? Did you check flip rate? It should be around ~97%. Also, make sure that you are running DiME in the complete set: i.e. use the flag --dataset CelebA and not --dataset CelebAMW.

Best,

Hi @guillaumejs2403,

Thanks for your quick response. Where can I check the flip rate? I made sure to use the flag --dataset CelebA. I used the exact same parameters mentioned in the ReadME. Please let me what else I can check.

Thanks

[guillaumejs2403]

Please, do the following:

1) Check the ${OUTPUT}/Results/${EXPNAME} folder. There should be a file called summary.yaml. Open it and look at the cf acc and n descriptors. They should be ~97 and ~20000, respectively.

• If this file does not exist, but there is a file called chunk-x_num-chunks-X_summary.yaml, then it means that you ran only a single chunk.
• If this file does exist, but the n descriptor is not the correct one, check the flag --num_batches. In the test.sh, I set that flag to 1, please change it for 20000. 2) How does the results look like? Qualitatively, are they good?

[sakethbachu]

Please, do the following:

1. Check the ${OUTPUT}/Results/${EXPNAME} folder. There should be a file called summary.yaml. Open it and look at the cf acc and n descriptors. They should be ~97 and ~20000, respectively.

   • If this file does not exist, but there is a file called chunk-x_num-chunks-X_summary.yaml, then it means that you ran only a single chunk.
   • If this file does exist, but the n descriptor is not the correct one, check the flag --num_batches. In the test.sh, I set that flag to 1, please change it for 20000.
2. How does the results look like? Qualitatively, are they good?

1. I have summary.yaml but the --num_batches is set to 1. Do you want me to set --num_batches to 20000? Setting it to 20000, its taking a lot of time for image generation.
2. Qualitatively, they do not look that good.

[sakethbachu]

@guillaumejs2403 Also, in the file 'celeba-train-diffusion.py', the line 'from guided_diffusion.image_datasets import load_data_celeba' throws an error as the underlying guided_diffusion repo that you mentioned in the ReadME does not have 'load_data_celeba' function in the image_datasets file. Can you please point me that missing file?

[guillaumejs2403]

Dear @sakethbachu

1) Yes, set --num_batches to 20000 2) Thanks for pointing that one out. I forgot to change the from guided_diffusion.image_datasets import load_data_celeba to from core.image_datasets import load_data_celeba.

[sakethbachu]

Dear @sakethbachu

1. Yes, set --num_batches to 20000
2. Thanks for pointing that one out. I forgot to change the from guided_diffusion.image_datasets import load_data_celeba to from core.image_datasets import load_data_celeba.

But setting --num_batches to 20000 is taking for ever to generate images. How much time it should take, approx?

[guillaumejs2403]

That is normal since we generate each explanation using 1800 DDPM iterations per image. If I remember correctly, generating all the explanations should take about 1 week using 4 GPUs - with NVIDIA 1080 GPUs. If you are interested in running DiME with multiple GPUs, please refer to the Extracting Counterfactual Explanations section.

[sakethbachu]

That is normal since we generate each explanation using 1800 DDPM iterations per image. If I remember correctly, generating all the explanations should take about 1 week using 4 GPUs - with NVIDIA 1080 GPUs. If you are interested in running DiME with multiple GPUs, please refer to the Extracting Counterfactual Explanations section.

Sorry for not stating my aim earlier here. I want to take an image x (which has an attribute of 'not smiling') and generate a counterfactual of it with the attribute 'smiling'. I used the sample generating script to do this but I want good FID scores as mentioned in the paper (i.e 3.17) but instead I got 55 as mentioned above in the discussion. Please let me know how can I achieve what I stated (with minimum time as possible).

[guillaumejs2403]

The FID metric is (very) sensible with respect to the number of images. Hence, only way to achieve this value is by running DiME over all the dataset. To speed up the generation using multiple GPUs, please follow the "Extracting Counterfactual Explanations" section. There is no other way to get this FID. I'll close this issue right now.

Best,