performance gap with the reported results using cuhk03 (L)

[jpainam]

Hi, i run the exact script run_rgasc_cuhk03.sh but the results are really different compared with the one you reported in the paper. I also had to skip layer1.0.bn1.num_batches_tracked and bn1.num_batches_tracked when loading the specific parameters. not sure if this could be the problem.

=> CUHK03 (labeled) loaded
Dataset statistics:
  ------------------------------
  subset   | # ids | # images
  ------------------------------
  train    |   767 |     7368
  query    |   700 |     1400
  gallery  |   700 |     5328
  ------------------------------
  total    |  1467 |     8768
  ------------------------------
Num of features: 2048.
Use_Spatial_Att: True;  Use_Channel_Att: True.
Use_Spatial_Att: True;  Use_Channel_Att: True.
Use_Spatial_Att: True;  Use_Channel_Att: True.
Use_Spatial_Att: True;  Use_Channel_Att: True.
=> Loaded checkpoint './logs/RGA-SC/cuhk03labeled_b64f2048/checkpoint_600.pth.tar'

Here are my results

Evaluated with "feat_" features and "cosine" metric:
Mean AP: 74.8%
CMC Scores
  top-1          77.0%
  top-5          89.9%
  top-10         94.4%
Evaluated with "feat" features and "cosine" metric:
Mean AP: 74.0%
CMC Scores
  top-1          75.9%
  top-5          89.1%
  top-10         93.4%

The paper reports a rank-1 accuracy of 81.1 while I get 77.0. This is a huge gap (4.1)

[longma2019]

I also encounter the same problem. The PyTorch version I used is 1.6.0. To avoid the KeyError: 'bn1.num_batches_tracked', I made some revisions of rga_branches.py:

def load_partial_param(self, state_dict, model_index, model_path):
    param_dict = torch.load(model_path)
    for i in state_dict:
        key = 'layer{}.'.format(model_index)+i
        **if 'num_batches_tracked' in key:
            continue**
        state_dict[i].copy_(param_dict[key])
    del param_dict

def load_specific_param(self, state_dict, param_name, model_path):
    param_dict = torch.load(model_path)
    for i in state_dict:
        key = param_name + '.' + i
        **if 'num_batches_tracked' in key:
            continue**
        state_dict[i].copy_(param_dict[key])
    del param_dict

[jpainam]

@longma2019 does it also affect your mAP and rank-1 accuracy? I get different results compared to the authors' results

[longma2019]

@longma2019 does it also affect your mAP and rank-1 accuracy? I get different results compared to the authors' results

Here is my result from checkpoint_600:

Evaluated with "feat_" features and "cosine" metric: Mean AP: 77.0% CMC Scores top-1 78.7% top-5 91.1% top-10 94.9% Evaluated with "feat" features and "cosine" metric: Mean AP: 75.6% CMC Scores top-1 76.7% top-5 89.4% top-10 94.3%

[jpainam]

our results are similar. thank you for sharing. the authors of this work have to tell us what we are doing wrong.

[gwenzhang]

In my experiment, the last epoch result is not the best. The best result is around 360 epoch. I think you need to test the middle epoch result.

[zhangcui2gr]

I got a worse result. Evaluated with "feat_" features and "cosine" metric: Mean AP: 36.7% CMC Scores top-1 41.6% top-5 63.6% top-10 72.8% Evaluated with "feat" features and "cosine" metric: Mean AP: 37.4% CMC Scores top-1 41.4% top-5 63.4% top-10 73.5%

Then it is my config file. I can not find other problems.

parser.add_argument('-d', '--dataset', type=str, default='cuhk03labeled')
parser.add_argument('-b', '--batch-size', type=int, default=32)
parser.add_argument('-j', '--workers', type=int, default=8)
parser.add_argument('--split', type=int, default=0)
parser.add_argument('--height', type=int,
                    help="input height, default: 256 for resnet*, "
                         "144 for inception")
parser.add_argument('--width', type=int,
                    help="input width, default: 128 for resnet*, "
                         "56 for inception")
parser.add_argument('--combine-trainval', action='store_true',
                    help="train and val sets together for training, "
                         "val set alone for validation")
parser.add_argument('--num-instances', type=int, default=4,
                    help="each minibatch consist of "
                         "(batch_size // num_instances) identities, and "
                         "each identity has num_instances instances, "
                         "default: 4")
# model
parser.add_argument('-a', '--arch', type=str, default='resnet50_rga',
                    choices=models.names())
parser.add_argument('--features', type=int, default=2048)
parser.add_argument('--dropout', type=float, default=0.5)
parser.add_argument('--branch_name', type=str, default='rgasc')
parser.add_argument('--use_rgb', type=str2bool, default=True)
parser.add_argument('--use_bn', type=str2bool, default=True)
# loss
parser.add_argument('--margin', type=float, default=0.3,
                    help="margin of the triplet loss, default: 0.3")
# optimizer
parser.add_argument('-opt', '--optimizer', type=str, default='sgd')
parser.add_argument('--lr', type=float, default=0.1,
                    help="learning rate of new parameters, for pretrained "
                         "parameters it is 10 times smaller than this")
parser.add_argument('--momentum', type=float, default=0.9)
parser.add_argument('--weight-decay', type=float, default=5e-4)
# training configs
parser.add_argument('--num_gpu', type=int, default=1)
parser.add_argument('--resume', type=str, default='', metavar='PATH')
parser.add_argument('--evaluate', action='store_true',
                    default=False, help="evaluation only")
parser.add_argument('--rerank', action='store_true',
                    help="evaluation with re-ranking")
parser.add_argument('--epochs', type=int, default=600)
parser.add_argument('--start_save', type=int, default=10,
                    help="start saving checkpoints after specific epoch")
parser.add_argument('--seed', type=int, default=16)
parser.add_argument('--print-freq', type=int, default=1)
parser.add_argument('--empty_cache', type=str2bool, default=False)
parser.add_argument('--random_erasing', type=str2bool, default=True)
# metric learning
parser.add_argument('--dist-metric', type=str, default='euclidean',
                    choices=['euclidean', 'kissme'])

[jpainam]

Hi, please, can you give us your execution environment (python version and pytorch). Also, make sure these Use_Spatial_Att and Use_Channel_Att are set to True, you can see that in the output when running. Confirm the statistics of the dataset also.

=> CUHK03 (labeled) loaded
Dataset statistics:
  ------------------------------
  subset   | # ids | # images
  ------------------------------
  train    |   767 |     7368
  query    |   700 |     1400
  gallery  |   700 |     5328
  ------------------------------
  total    |  1467 |     8768
  ------------------------------
Num of features: 2048.
Use_Spatial_Att: True;  Use_Channel_Att: True.
Use_Spatial_Att: True;  Use_Channel_Att: True.
Use_Spatial_Att: True;  Use_Channel_Att: True.
Use_Spatial_Att: True;  Use_Channel_Att: True.

What about the .sh inside the script folder?

[zhangcui2gr]

Hi, please, can you give us your execution environment (python version and pytorch). Also, make sure these Use_Spatial_Att and Use_Channel_Att are set to True, you can see that in the output when running. Confirm the statistics of the dataset also.

=> CUHK03 (labeled) loaded
Dataset statistics:
  ------------------------------
  subset   | # ids | # images
  ------------------------------
  train    |   767 |     7368
  query    |   700 |     1400
  gallery  |   700 |     5328
  ------------------------------
  total    |  1467 |     8768
  ------------------------------
Num of features: 2048.
Use_Spatial_Att: True;    Use_Channel_Att: True.
Use_Spatial_Att: True;    Use_Channel_Att: True.
Use_Spatial_Att: True;    Use_Channel_Att: True.
Use_Spatial_Att: True;    Use_Channel_Att: True.

What about the .sh inside the script folder?

python3.6.10 pytorch 0.4.0 The statistics of the dataset is same as you. Use_Spatial_Att and Use_Channel_Att is alse same. The training part in .sh inside the script folder has a difference with the config: --opt sgd in the config --opt adam in the .sh Dose it make a great influence in taining?

[jpainam]

Not sure if using --opt sgd will have such a big impact. But I trained mine using the scripts file with adam. you may that.

bash scripts/run_rgasc_cuhk03.sh

11 also reported very low results. It's possible that you do multiples training. Maybe sometimes it converges, other times it's stuck at a local minimum.

Evaluated with "feat_" features and "cosine" metric:
Mean AP: 33.2%
CMC Scores
  top-1          30.7%
  top-5          56.6%
  top-10         69.6%
Evaluated with "feat" features and "cosine" metric:
Mean AP: 21.4%
CMC Scores
  top-1          17.7%
  top-5          36.8%
  top-10         52.3%

[zhangcui2gr]

Not sure if using --opt sgd will have such a big impact. But I trained mine using the scripts file with adam. you may that.

bash scripts/run_rgasc_cuhk03.sh

11 also reported very low results. It's possible that you do multiples training. Maybe sometimes it converges, other times it's stuck at a local minimum.

Evaluated with "feat_" features and "cosine" metric:
Mean AP: 33.2%
CMC Scores
  top-1          30.7%
  top-5          56.6%
  top-10         69.6%
Evaluated with "feat" features and "cosine" metric:
Mean AP: 21.4%
CMC Scores
  top-1          17.7%
  top-5          36.8%
  top-10         52.3%

I have change the --opt sgd to --opt adam Then it comes to look a bit normal like you

feat_name: feat_
feat_name: feat
Evaluated with "feat_" features and "cosine" metric:
Mean AP: 71.1%
CMC Scores
  top-1          75.0%
  top-5          88.6%
  top-10         93.1%
Evaluated with "feat" features and "cosine" metric:
Mean AP: 65.7%
CMC Scores
  top-1          70.0%
  top-5          84.4%
  top-10         90.7%

And I will read the link which you said to optimize. Thank you very much!!!