Testing result on Market1501

[Ken2yLiu]

Hi,

I download your pre-trained model and test it on Market1501 dataset following your README which using the default config of train_pcb.py with "only_test" parameter, however, the result scores are lower than that in your README release table. The logs are: =========> Test on dataset: market1501 <=========

Extracting feature... 1000/1000 batches done, +2.88s, total 139.10s Done, 139.30s Computing distance... Done, 2.00s Computing scores... Done, 10.08s Single Query: [mAP: 64.96%], [cmc1: 88.66%], [cmc5: 94.95%], [cmc10: 96.35%] Multi Query, Computing distance... Done, 1.99s Multi Query, Computing scores... Done, 10.06s Multi Query: [mAP: 73.32%], [cmc1: 91.69%], [cmc5: 96.53%], [cmc10: 97.68%] Re-ranking distance... Done, 73.66s Computing scores for re-ranked distance... Done, 10.90s Re-ranked Single Query: [mAP: 85.22%], [cmc1: 91.57%], [cmc5: 94.98%], [cmc10: 96.20%] Multi Query, Re-ranking distance... Done, 65.32s Multi Query, Computing scores for re-ranked distance... Done, 10.88s Re-ranked Multi Query: [mAP: 88.03%], [cmc1: 92.99%], [cmc5: 96.17%], [cmc10: 96.79%]

[huanghoujing]

Thanks for your attention.

I clone the code to a new machine and run the following script (after processing dataset etc)

python script/experiment/train_pcb.py \
-d '(0,)' \
--only_test true \
--dataset market1501 \
--exp_dir exp/test-market1501-independent-1x1conv \
--model_weight_file exp/test-market1501-independent-1x1conv/model_weight.pth

The scores are identical to README file, as follows

----------------------------------------
market1501 test set
----------------------------------------
NO. Images: 31969
NO. IDs: 751
NO. Query Images: 3368
NO. Gallery Images: 15913
NO. Multi-query Images: 12688
----------------------------------------
Loaded model weights from exp/test-market1501-independent-1x1conv/model_weight.pth

=========> Test on dataset: market1501 <=========

Extracting feature...
1000/1000 batches done, +3.05s, total 156.72s
Done, 157.15s
Computing distance...
Done, 1.25s
Computing scores...
Done, 14.37s
Single Query:                 [mAP: 78.54%], [cmc1: 92.87%], [cmc5: 96.88%], [cmc10: 98.07%]
Multi Query, Computing distance...
Done, 1.32s
Multi Query, Computing scores...
Done, 14.43s
Multi Query:                  [mAP: 84.68%], [cmc1: 95.10%], [cmc5: 98.04%], [cmc10: 98.75%]
Re-ranking distance...
Done, 94.67s
Computing scores for re-ranked distance...
Done, 15.69s
Re-ranked Single Query:       [mAP: 90.17%], [cmc1: 93.94%], [cmc5: 96.62%], [cmc10: 97.24%]
Multi Query, Re-ranking distance...
Done, 89.70s
Multi Query, Computing scores for re-ranked distance...
Done, 15.81s
Re-ranked Multi Query:        [mAP: 93.02%], [cmc1: 95.81%], [cmc5: 97.62%], [cmc10: 98.07%]

Do you use pytorch 0.3 and python 2.7? For other packages used, you can try to install the versions in requirements.txt.

[Ken2yLiu]

Thanks for your reply.

It seems I used the model_weight downloaded on 3 Apr which is different with the one you use now. Then I download the latest model in the Baidu Drive you provided and get the same result as yours.

However, I am confuse with why the old model weight can still be loaded, since I have clone your latest version code, which according to your commit, is fixed to support independent 1x1 conv. As I know, the model in earlier version should use shared 1x1 conv, they must have different network structures.

[huanghoujing]

Well, mismatch between source (loaded from disk) and destination (currently defined in code) state_dicts is considered in my code, see here and here. The mismatch would cause warnings, not errors.

In your case, some 1x1 conv layers are defined in the code but not found in the loaded state dict, so you would see Keys not found in source state_dict: in your log file.

[Ken2yLiu]

Well, I found that in logs. And If I use the shared 1x1 conv model, the test result will be different each time, and always lower than that you release, which should caused by the missing 1x1 conv parameters initialized randomly.

[huanghoujing]

Yes, I think it's the reason.