Poor performnace when reproducing evaluation on market1501

[rrryan2016]

Thanks for your great work and kind sharing.

I am actually a beginner in ReID, and trying to reproduce the test stage of this repo at first.

Environment: 3090Ti, CUDA 11.0, python 2.7, pytorch 0.3.1

I strictly follow the guidance on README, but get below result for ResNet-50 + Global Loss on Market1501

Extracting feature... 1000/1000 batches done, +0.58s, total 454.75s Done, 454.93s Computing global distance... Done, 0.57s Computing scores for Global Distance... [mAP: 0.11%], [cmc1: 0.06%], [cmc5: 0.53%], [cmc10: 0.95%] Done, 8.84s Re-ranking... Done, 52.41s Computing scores for re-ranked Global Distance... [mAP: 0.11%], [cmc1: 0.06%], [cmc5: 0.53%], [cmc10: 0.95%] Done, 9.68s

In detail, I downloaded transformed Market1501 by the google drive link you provided, and configure corresponding codes in __init__.py as stated in https://github.com/huanghoujing/AlignedReID-Re-Production-Pytorch#configure-dataset-path.

Then, I downloaded saved model weights of both ResNet-50 + Global Loss with or without mutual learning in the google drive link.

Here is the detailed command and result for ResNet-50 + Global Loss + Mutual Learning on Market1501 setting:

python script/experiment/train.py \ -d '(0,)' \ --dataset market1501 \ --normalize_feature false \ -glw 1 \ -llw 0 \ -idlw 0 \ --only_test true \ --exp_dir /home/vgc/users/lwz/code/reID/AlignedReID-Re-Production-Pytorch/results/exp/resnet50_global_loss_mutual \ --model_weight_file /home/vgc/users/lwz/result/reID/checkpoint/ResNet-50_Global_Loss_Mutual_Learning/model_weight.pth

and all the output

cfg.dict {'base_lr': 0.0002, 'ckpt_file': '/home/vgc/users/lwz/code/reID/AlignedReID-Re-Production-Pytorch/results/exp/resnet50_global_loss_mutual/ckpt.pth', 'crop_prob': 0, 'crop_ratio': 1, 'dataset': 'market1501', 'exp_decay_at_epoch': 76, 'exp_dir': '/home/vgc/users/lwz/code/reID/AlignedReID-Re-Production-Pytorch/results/exp/resnet50_global_loss_mutual', 'g_loss_weight': 1.0, 'global_margin': 0.3, 'id_loss_weight': 0.0, 'ids_per_batch': 32, 'im_mean': [0.486, 0.459, 0.408], 'im_std': [0.229, 0.224, 0.225], 'ims_per_id': 4, 'l_loss_weight': 0.0, 'local_conv_out_channels': 128, 'local_dist_own_hard_sample': False, 'local_margin': 0.3, 'log_steps': 10000000000.0, 'log_to_file': True, 'lr_decay_type': 'exp', 'model_weight_file': '/home/vgc/users/lwz/result/reID/checkpoint/ResNet-50_Global_Loss_Mutual_Learning/model_weight.pth', 'normalize_feature': False, 'only_test': True, 'prefetch_threads': 2, 'resize_h_w': (256, 128), 'resume': False, 'run': 1, 'scale_im': True, 'seed': None, 'staircase_decay_at_epochs': (101, 201), 'staircase_decay_multiply_factor': 0.1, 'stderr_file': '/home/vgc/users/lwz/code/reID/AlignedReID-Re-Production-Pytorch/results/exp/resnet50_global_loss_mutual/stderr_2021-06-12_15:42:23.txt', 'stdout_file': '/home/vgc/users/lwz/code/reID/AlignedReID-Re-Production-Pytorch/results/exp/resnet50_global_loss_mutual/stdout_2021-06-12_15:42:23.txt', 'sys_device_ids': (1,), 'test_batch_size': 32, 'test_final_batch': True, 'test_mirror_type': None, 'test_set_kwargs': {'batch_dims': 'NCHW', 'batch_size': 32, 'final_batch': True, 'im_mean': [0.486, 0.459, 0.408], 'im_std': [0.229, 0.224, 0.225], 'mirror_type': None, 'name': 'market1501', 'num_prefetch_threads': 2, 'part': 'test', 'prng': <module 'numpy.random' from '/home/vgc/anaconda3/envs/lwz27/lib/python2.7/site-packages/numpy/random/init.pyc'>, 'resize_h_w': (256, 128), 'scale': True, 'shuffle': False}, 'test_shuffle': False, 'total_epochs': 150, 'train_final_batch': False, 'train_mirror_type': 'random', 'train_set_kwargs': {'batch_dims': 'NCHW', 'crop_prob': 0, 'crop_ratio': 1, 'final_batch': False, 'ids_per_batch': 32, 'im_mean': [0.486, 0.459, 0.408], 'im_std': [0.229, 0.224, 0.225], 'ims_per_id': 4, 'mirror_type': 'random', 'name': 'market1501', 'num_prefetch_threads': 2, 'part': 'trainval', 'prng': <module 'numpy.random' from '/home/vgc/anaconda3/envs/lwz27/lib/python2.7/site-packages/numpy/random/init.pyc'>, 'resize_h_w': (256, 128), 'scale': True, 'shuffle': True}, 'train_shuffle': True, 'trainset_part': 'trainval', 'weight_decay': 0.0005}

---

market1501 trainval set

NO. Images: 12936 NO. IDs: 751

---

market1501 test set

NO. Images: 31969 NO. IDs: 751 NO. Query Images: 3368 NO. Gallery Images: 15913 NO. Multi-query Images: 12688

/home/vgc/anaconda3/envs/lwz27/lib/python2.7/site-packages/torch/cuda/init.py:95: UserWarning: Found GPU0 GeForce RTX 3090 which requires CUDA_VERSION >= 9000 for optimal performance and fast startup time, but your PyTorch was compiled with CUDA_VERSION 8000. Please install the correct PyTorch binary using instructions from http://pytorch.org

warnings.warn(incorrect_binary_warn % (d, name, 9000, CUDA_VERSION)) Loaded model weights from /home/vgc/users/lwz/result/reID/checkpoint/ResNet-50_Global_Loss_Mutual_Learning/model_weight.pth

=========> Test on dataset: market1501 <=========

Extracting feature... 1000/1000 batches done, +0.58s, total 458.88s Done, 459.08s Computing global distance... Done, 0.60s Computing scores for Global Distance... [mAP: 1.63%], [cmc1: 0.06%], [cmc5: 0.45%], [cmc10: 0.92%] Done, 8.12s Re-ranking... Done, 53.22s Computing scores for re-ranked Global Distance... [mAP: 1.63%], [cmc1: 0.06%], [cmc5: 0.45%], [cmc10: 0.92%] Done, 8.62s

Any recommendation for what to do next? :P

Thanks in advance.

[rrryan2016]

After I training on Market1501, the evaluation results turn to be,

=========> Test on dataset: market1501 <========= Extracting feature... 1000/1000 batches done, +0.58s, total 30.42s Done, 30.58s Computing global distance... Done, 0.52s Computing scores for Global Distance... [mAP: 50.04%], [cmc1: 0.00%], [cmc5: 0.24%], [cmc10: 0.24%] Done, 5.63s Re-ranking... Done, 39.60s Computing scores for re-ranked Global Distance... [mAP: 50.04%], [cmc1: 0.00%], [cmc5: 0.24%], [cmc10: 0.24%] Done, 6.17s

[rrryan2016]

If I change the renset50 to Densenet121, and do training, it comes out like,

Extracting feature... 1000/1000 batches done, +0.77s, total 39.60s Done, 39.70s Computing global distance... Done, 0.45s Computing scores for Global Distance... [mAP: 50.05%], [cmc1: 0.00%], [cmc5: 0.24%], [cmc10: 0.24%] Done, 5.92s Re-ranking... ./aligned_reid/utils/re_ranking.py:45: RuntimeWarning: invalid value encountered in divide original_dist = np.transpose(1. * original_dist/np.max(original_dist,axis = 0)) Done, 42.30s