eval plot transfer difference

[shinyke]

Hi, I'm tring to evaluate model by eval.py and confusing why the test_tranform missing 'ColorJitter' step?

test_transform = transforms.Compose([])
# why missing this part in test phase.
# test_transform.transforms.append(transforms.ColorJitter(brightness=0.1, contrast=0.1, saturation=0.1, hue=0.1))
test_transform.transforms.append(transforms.Resize((size, size)))
test_transform.transforms.append(transforms.ToTensor())
test_transform.transforms.append(transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                                      std=[0.229, 0.224, 0.225]))

In training phase:

  after_cutpaste_transform = transforms.Compose([])
  after_cutpaste_transform.transforms.append(transforms.ToTensor())
  after_cutpaste_transform.transforms.append(transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                                                  std=[0.229, 0.224, 0.225]))
  train_transform = transforms.Compose([])

  train_transform.transforms.append(transforms.ColorJitter(brightness=0.1, contrast=0.1, saturation=0.1, hue=0.1))
  train_transform.transforms.append(transforms.Resize((size, size)))
  train_transform.transforms.append(cutpate_type(transform = after_cutpaste_transform))

[Runinho]

You might want to experiment with color jitter in the evaluation phase. But it will make the evaluation non-determinant.

My motivation to not use ColorJitter during evaluation was the following: ColorJitter is used during training to prevent the model from over fitting. We want that the model is working on an input space as big as possible. In the evaluation phase, we calculate the representations we fit the Gaussian mixture model on. We want that the good data lies close together and far away from the defect data points. ColorJitter might cause the good data to be more scattered and therefore decrease the performance on the benchmark.

Let me know if you did some experiments with and without ColorJitter and if my explanation helped you.

[shinyke]

Thx for nice explanation!

According to my current experimental results, using ColorJitter in the test stage will lead to the decline of AUC, which may be caused by the reasons you explained.