search

seominseok0429 / Real-world-Anomaly-Detection-in-Surveillance-Videos-pytorch

Real-world Anomaly Detection in Surveillance Videos- pytorch Re-implementation
114 stars 26 forks source link

Evaluation Strategy Is Incorrect #14

Closed pmcb99 closed 2 years ago

pmcb99 commented 2 years ago

You cannot calculate the AUC score in this way, as it leads to a higher overall AUC in the end.

Since you increment the AUC for every new addition of a video pair, the early increments are closer to 1 than 0, which inflates the final AUC. If anyone wishes to test this, use a small subset of the dataset and you will see very high AUC scores.

seominseok0429 commented 2 years ago

Compare with the code in the link: https://github.com/junha-kim/Learning-to-Adapt-to-Unseen-Abnormal-Activities/blob/master/evaluation.py#L27-L128

pmcb99 commented 2 years ago

I have already compared the code and it is wrong. You need to verify the code independently.

seominseok0429 commented 2 years ago

At the time of writing the code, it was confirmed that the performance was similar to the evaluation code of the link.

It's been a long time and it's difficult to accurately analyze the code.

Can you edit the code and commit it? Then I will merge.