search

StevenLiuWen / ano_pred_cvpr2018

Official implementation of Paper Future Frame Prediction for Anomaly Detection -- A New Baseline, CVPR 2018
435 stars 126 forks source link

Questions about evaluation schemes used in the paper #17

Closed AdarshMJ closed 5 years ago

AdarshMJ commented 5 years ago

Question1 I had few questions on the evaluation schemes provided in the paper. The PSNR evaluation scheme says that you have to calculate the regularity score by using the PSNR values of every test frame and then set a threshold. If my test video changes will the same threshold work? Can the same threshold value be used for similar type of videos?

Question2 In the paper where AUC of different methods is given. Could you please share how exactly this is calculated?

Question3 In the section of paper you describe "We first compute the gap between average score of normal frames and that of abnormal frames, denoted as ∆s". What is average score here?

zachluo commented 5 years ago

@AdarshMJ Question 1: We use AUC/ROC which means we don't set a specific threshold in this paper. You can select a threshold from a evaluation set and use it in any testing videos.

Question 2: We use sklearn package to calculate AUC. After normalizing PSNR into normal scores [0, 1], AUC will use, for example, 10 thresholds (0, 0.1, 0.2, ..., 0.9, 1.0) and for each specific threshold, one can calculate TPR, FPR, etc. Finally, all results are formed into a AUC value.

Question3: It is normal scores normalizede from PSNR of normal and abnormal frames. We wanna show the gap between normal scores of normal frames and normal scores of abnormal frames.