Seeking advise for unreasonable results in my test

[hothothot2000]

First, thanks a lot for you excellent work. I used your code with only a little adaption mainly in _evaluate() in engine.py, so that I can do test & watch results more convenient. While almost with no modification to all other py files, my _evaluate() looks like following:

    DIR_GALLERY  = './reid-data/pics_in/gallery'
    DIR_QUERY    = './reid-data/pics_in/query'
    imgsQ, fnamesQ = _imread_from_dir(DIR_QUERY)
    ftrsQ          = _get_features_from_imgs(imgsQ)
    imgsG, fnamesG = _imread_from_dir(DIR_GALLERY)
    ftrsG          = _get_features_from_imgs(imgsG)   
    distmat = metrics.compute_distance_matrix(ftrsQ, ftrsG, dist_metric)    
    distmat = distmat.numpy()
    _print_distmat(distmat, fnamesQ, fnamesG)

I used 11 jpgs as gallery. In the gallery, there are 4 persons whose jpg files are corresponding to 3X.jpg, 4X.jpg, 5X.jpg, 6X.jpg. (X is one digit substitution)

I used 4 jpgs as query, which are choosed from above gallery. (each person got a jpg: 31.jpg, 41.jpg, 51.jpg, 61.jpg)

I run test on ubuntu 16.04 with py3.5, pytorch 1.2.0-nightly, opencv-python 3.4.5. And I got a very unreasonable result of 'distmat'-- it looks that same person's images didn't achieve good distances.

Would anyone please give me any advise or tips on this result?

In this result, rectangles with the same color belong to the same person's comparison. And, it looks that same person's pics didn't get smaller(but much bigger) distances, even these 2 pics are obviously similar to each other, for ex: 61 & 62, as well as, 31 & 33.

[KaiyangZhou]

it seems like the network is not well learned, how did you train the model?

as we don't know much about your work, it's hard to see where it went wrong

[hothothot2000]

I didn't expect such quick an answer in these Spring Festival Days, thank you so much and happy chinese new year to you.

I tested with 'osnet_x1_0' which is the original model file downloaded from your link https://mega.nz/#!YK5GRARL!F90NsNB2XHjXGZFC3Lrw1GMic0oMw4fnfuDUnSrPAYM.

And I didn't do any train on this model file beacuse I thought it had been trained a lot in your research work which should be enough for my small test. I used dist_metric='euclidean'.

[KaiyangZhou]

can you do a sanity check by testing the downloaded model osnet_x1_0 on the corresponding dataset? Just to make sure the performance is the same as what is reported in the model zoo.

what's inside _get_features_from_imgs ()?

what does the distance matrix look like if you use dist_metric=cosine

[hothothot2000]

My _get_features_from_imgs(), defined in _evaluate() as following which is almost the same ops as original:

    def _get_features_from_imgs(imgsIn):
        imgs = torch.from_numpy(imgsIn).cuda()
        ftrs = self._extract_features(imgs)   
        return ftrs.data.cpu()

---

performances listed as following: (--1--). When no train at all. (1a).Tested on Market1501 with dist_metric='euclidean': mAP: 3.9% Rank-1 : 13.2% Rank-5 : 26.0% Rank-10 : 33.3% Rank-20 : 40.3%

(1b).Tested on Market1501 with dist_metric='cosine': mAP: 4.4% Rank-1 : 14.5% Rank-5 : 27.8% Rank-10 : 34.3% Rank-20 : 41.8%

(--2--). After trained with 100 epoch on Market1501, (2a).Tested on Market1501 with dist_metric='euclidean' mAP: 83.5% Rank-1 : 94.3% Rank-5 : 97.9% Rank-10 : 98.7% Rank-20 : 99.3%

(2b).Tested on Market1501 with dist_metric='cosine' mAP: 85.0% Rank-1 : 94.6% Rank-5 : 97.8% Rank-10 : 98.7% Rank-20 : 99.2% #################### And, by printing distmat on condition (2a) as following, I can see some improvements. Later I would train longer to expect better effect, or, if you have further suggestion.

[KaiyangZhou]

did you directly apply the model trained on Market1501 to your dataset? if so, it's common that the performance is low (due to the domain shift)

one more thing: is the test image size the same as the training image size (say 256x128)?

for direct transfer, I'd suggest you use our latest model osnet_ain_x1_0 (use msmt17 for pre-training, the model weights are available for download in the model zoo)

see our paper for more details

[hothothot2000]

I see, I didn't read details of second half of the paper before I rushed to test for a glimps on the actual effect. I downloaded the AIN model. I think this AIN model also need some train before test.

But I couldn't train before I got a second dataset, an then I found these links for datasets are broken: http://vision.cs.duke.edu/DukeMTMC/ http://www.pkuvmc.com/publications/msmt17.html https://users.soe.ucsc.edu/~manduchi/VIPeR.v1.0.zip

Then, I managed to find another Baiduyun link for DukeMTMC https://pan.baidu.com/s/1jS0XM7Var5nQGcbf9xUztw