search

VISION-SJTU / RECCE

[CVPR2022] End-to-End Reconstruction-Classification Learning for Face Forgery Detection
MIT License
103 stars 14 forks source link

Unexpected performance drop when not using provided test code. #5

Closed deepfakeLearnerHi closed 1 year ago

deepfakeLearnerHi commented 2 years ago

Reproducing with a bit more data (20 frames per video), I got a fairly good checkpoint. However, when I evaluated the checkpoint without the provided test code, the performance dropped unexpected. I wonder if I did something wrong. Here's what I did:

  1. Load my test data
  2. Perform augmentation: images normalized with mean an std both [0.5, 0.5, 0.5]; images resized to (299, 299)
  3. Labels: Real=0, Fake=1
  4. Put all data to dataloader
  5. Feed all data from dataloader to the model and calculate the ACC and AUC.

Please let me know if I am missing any important step. And it will be very helpful if a well-trained checkpoint could be provided.

Blosslzy commented 2 years ago

Hi, can your good checkpoint get the same results as shown in the paper?

XJay18 commented 2 years ago

Hi, actually I cannot find any important missing steps based on the description. I suggest modifying the provided test code line by line to find the reason.

deepfakeLearnerHi commented 1 year ago

Found the problem although I don't know why: I did augmentations using the torchvision.transforms, which led to the bad results. However, when I used the augmentation which the source code did, it worked just fine.

I think your source code is good though, something must be wrong on my side. Thanks for your time!

XJay18 commented 1 year ago

Hi, thanks for reporting the reason. I guess it may result from the different implementations of them. Best wishes

xarryon commented 1 year ago

Reproducing with a bit more data (20 frames per video), I got a fairly good checkpoint. However, when I evaluated the checkpoint without the provided test code, the performance dropped unexpected. I wonder if I did something wrong. Here's what I did:

  1. Load my test data
  2. Perform augmentation: images normalized with mean an std both [0.5, 0.5, 0.5]; images resized to (299, 299)
  3. Labels: Real=0, Fake=1
  4. Put all data to dataloader
  5. Feed all data from dataloader to the model and calculate the ACC and AUC.

Please let me know if I am missing any important step. And it will be very helpful if a well-trained checkpoint could be provided.

Hi, could u share the detection performance you get with more frames? Thanks!