search

marco-rudolph / differnet

This is the official repository to the WACV 2021 paper "Same Same But DifferNet: Semi-Supervised Defect Detection with Normalizing Flows" by Marco Rudolph, Bastian Wandt and Bodo Rosenhahn.
215 stars 67 forks source link

Should the test data contain rotation tranform when calculating anomaly_score? #4

Closed jinzishuai closed 4 years ago

jinzishuai commented 4 years ago

Hi, there,

I found in the code, the evaluation takes two steps

  1. calculate z = model(inputs), loss and anomaly_score
  2. use export_gradient_maps to get the gradient map for locationzation.

It seem that the inputs used in the two cases are slightly different:

  1. in the first case, the inputs are just the original images after reizing
  2. in the second case, there is an additional step to add fixed rotations according to code in https://github.com/marco-rudolph/differnet/blob/master/multi_transform_loader.py#L49 since we set testloader.dataset.get_fixed = True in https://github.com/marco-rudolph/differnet/blob/master/localization.py#L38

The consequence is that the data used to calculate anomaly_socre are not averaged out among different rotations while the gradients are. Is this a problem? Shouldn't we be consistent and add the rotation tranformation to even the first step, ie, set testloader.dataset.get_fixed = True always? Maybe there is a reason to do this differently by design?

Thank you very much

marco-rudolph commented 4 years ago

Let me clear up the difference between 1) and 2): In 1) the pictures are already rotated randomly. As you can see here the same transformations as in training are used. The only difference to 2) is that there the rotations are not random, but the angles are fixed (regular steps from 0 to 360 degrees). Otherwise one would not know how to retransform the randomly rotated gradient map. Example for 2) and n_transforms_test = 4: Fixed rotation angles are [0, 90, 180, 270].

jinzishuai commented 4 years ago

You are absolutely right. Thank you very much for the clarification.

hoangphucITJP commented 4 years ago

In 1) the pictures are already rotated randomly. As you can see here the same transformations as in training are used.

Hi @marco-rudolph . I don't understand why you use random rotation for the test dataset. Shouldn't test set output be deterministic, or did I miss something?

marco-rudolph commented 4 years ago

I have found that it doesn't really make a difference whether the rotations are random or deterministic when testing - especially with 64 transformations.