cross testing on celeb-DF

[SaharHusseini]

Hello, First, thank you for your excellent work; I am testing your model using by Celeb-DF dataset. I used your provided model (FF++ c40) to test on celeb-DF. I cropped the celeb-DF dataset using code #1 and then used your data loader and test.py script for testing. However, the results I am getting are different from the paper. Can you advise me if there is something wrong?

Thank you for your help, and waiting for your answer

[XJay18]

Hi, I just tested the provided FF++ c40 model on Celeb-DF and the results are

Test 314/314: : 314it [03:30,  1.49it/s]
Test, FINAL LOSS 0.8964, FINAL ACC 0.6793, FINAL AUC 0.6871
# EER: 0.3575(thresh: 0.9161)

There are several reasons that may cause the difference:

1. please make sure that you are using Celeb-DF v2 dataset published at CVPR 2020,
2. the hyperparameters used during pre-preprocessing. Unfortunately, I am not able to retrieve the specific parameter settings for pre-processing because it's been too long since the preprocessing task. However, the following information may be helpful. The test split of Celeb-DF we used contains 7,114 real frames and 12,939 fake frames. You may adjust the frame interval during pre-processing.
3. the platform and the pytorch version. Our code is tested on 2080Ti with pytorch 1.7.1.

[SaharHusseini]

Hello,

Thanks for your response. Could you please explain what you mean by specific parameter settings for preprocessing? I am using the code provided in #1 for the preprocessing. Do we need to apply another kind of preprocessing? Is the preprocessing the same for all datasets?
thank you in advance for your help

[XJay18]

Hi, I mean the hyperparameter --frame_interval in #1. We use the same preprocessing code to detect and crop facial images for the video-based datasets. However, the frame interval is set differently since these datasets vary in size.

If you need our preprocessed Celeb-DF dataset, you may access it through this link (Please note that this link is temporary since I do not have enough sharing space and may delete these files for space in the future).

[SaharHusseini]

Hello,

Thank you very much, and I appreciate your responsiveness. As I am new to deepfake detection, your answers greatly help me.

I changed the interval, and below are the results:

So I get the accuracy you have in the paper. However, I wonder how different methods are compared with each other in your paper. Did you retest other methods with your preprocessed dataset or the values are just taken from the papers?

BR, Sahar

[XJay18]

Hi, For intra-testing, the results of other methods are taken from their papers if available, following the common practice of prior arts. However, some methods do not include the results on several datasets (e.g., DFDC). In this case, we re-implement some representative approaches according to their open-source repos or papers, and then run the experiments using our preprocessed data for comparison. For cross-testing, since different methods may use different training sets (e.g., raw, c23, or c40 of FF++) in their original papers, basically we compare our method with the re-implemented approaches using our preprocessed data for a fair comparison.

Best wishes

[SaharHusseini]

Thank you

[lijichang]

Hi @XJay18 @SaharHusseinim . Thanks for the enlightening discussion that you recently facilitated. It was indeed highly beneficial, and I garnered much from it.

In the course of the discussion, the previously shared link for the preprocessed Celeb-DF dataset was made. However, I have encountered a challenge with accessing this dataset as the link previously shared seems to have become invalid.

Consequently, I was wondering if it would be possible for you to re-share the link to the preprocessed Celeb-DF dataset @XJay18 . Or for @SaharHusseini , if you have the received contents readily available, I would greatly appreciate it if you could share them with me.

Your assistance in this matter will be highly beneficial to my ongoing research efforts. Thank you in advance for your kind consideration and help.

[jesslos]

Hello,

Thank you very much, and I appreciate your responsiveness. As I am new to deepfake detection, your answers greatly help me.

I changed the interval, and below are the results:

So I get the accuracy you have in the paper. However, I wonder how different methods are compared with each other in your paper. Did you retest other methods with your preprocessed dataset or the values are just taken from the papers?

BR, Sahar

Hi,

I recently had a problem with this part when testing the celeb-df dataset. The preprocessed Celeb-DF dataset shared by the author before has expired. Can you share the one you received? My email is 997834890@qq.com.

Best regards