Performance on Cars196 dataset

[m990130]

Hi, thanks for this great repo! I've tried out a few runs, and they work nicely.

I've also tested this method on the Cars196 dataset (with the same setup as CUB, I also wrote a dataset file for it, but almost the same). However, it performed pretty badly, with R@1=52%.

As it is one of the most evaluated datasets in deep metric learning community, I wonder if you have any idea why this is the case. Because usually if the methods work on CUB and SOP, they at least perform comparably on Cars196, and this is not the case. Thanks in advance.

[jhgan00]

In my experience, the sampling strategy had a huge impact on performance. I'm not sure, but it might be helpful to apply other sampling strategies such as m per class sampling(try something like python main.py --m 4 ...). Increasing the batch size can also be helpful ...

Can you share me your dataset code and training scripts? Maybe we can find some hyperparameters that work fine for Cars.

Thanks.

[m990130]

Hi, thanks for the quick response. Because the results are already quite good (compared to those results achieved by conventional ConvNet), I did not pay much attention to other hyper params. Indeed, batch-size and m per class are crucial.

• Here are my snippet of the cars dataset, feel free to add it to the repo:

  import os
  import numpy as np
  from torchvision import datasets
  from .base_dataset import BaseDataset
  
  class CARS196(BaseDataset):
  
      def __init__(self, *args, **kwargs):
          super(CARS196, self).__init__(*args, **kwargs)
          assert self.split in {"train", "test"}
  
      def set_paths_and_labels(self):
  
          dataset = datasets.ImageFolder(os.path.join(self.data_dir, 'images'))
          paths = np.array([a for (a, b) in dataset.imgs])
          labels = np.array([b for (a, b) in dataset.imgs])
          sorted_lb = list(sorted(set(labels)))
          if self.split == "train":
              set_labels = set(sorted_lb[:len(sorted_lb) // 2])
          else:
              set_labels = set(sorted_lb[len(sorted_lb) // 2:])
          self.paths = []
          self.labels = []
          for lb, pth in zip(labels, paths):
              if lb in set_labels:
                  self.paths.append(pth)
                  self.labels.append(lb)

  The dataset structure is the same as here, https://github.com/Confusezius/Revisiting_Deep_Metric_Learning_PyTorch, which basically uses the first half for training and the latter half for testing.

• For training, I used the same training script as in train.cub.sh but with modified dataset and data-path. Interestingly, by setting lambda-reg from 0.7 to 0 brings the R@1 score from 52% to ~86% .