search

PeterWang512 / CNNDetection

Code for the paper: CNN-generated images are surprisingly easy to spot... for now https://peterwang512.github.io/CNNDetection/
Other
851 stars 176 forks source link

CNN detection reports moderate performance on self-synthesized #14

Closed hadzica closed 3 years ago

hadzica commented 3 years ago

Hi all,

FIrst of all, thanks for providing your great work. My team and I are working on StyleGan to improve its synthesis. We want to benchmark our improvement with your CNNDetection. In a first step we synthesized 10k images from an StyleGan provided by https://github.com/genforce/genforce by using their pretrained models. By applying weights/blur_jpg_prob0.1.pth we report 72.94% fake detection accuracy for a StyleGan model which synthesizes images at 1024x1024. For 256x256 synthesis we report an even lower fake detection accuracy 65.36%. Can you maybe reproduce that and do you maybe have an idea what we do wrong in our workflow. Did we miss out a crucial step to make CNNDetection work?

Thanks on behalf of my team,

Amir

PeterWang512 commented 3 years ago

To understand this better, would be great to know what is your test images for real labels?

In our work, we are testing in a controlled scenario where the real set will be the exact dataset used to train the generative models. In other words, if a GAN resizes and crops training images from dataset A during training, we do resizing and cropping on dataset A to get our real images against that method.

Also, would also be great to know what the average precision is. We found out our model can be miscalibrated despite having a good average precision, so the accuracy can be not as ideal from time to time. This is something our work needs to improve on. However, we didn't observe a large miscalibration from styleGAN as far as I know.

Trying out the blur_jpg_prob0.5.pth model will also be great, we observed a better generalization performance for GANs with that model.

hadzica commented 3 years ago

Hi,

We observed that the reason for the bad performance was the image type. After synthesis we saved the images as jpg. Comparing with the online dataset we noticed that these are provided as png. After re-synthesizing we saved the images as png and we can report that the cnn detection works in that case.

Thanks for your help.