Market1501 use fixed query images

[luzai]

Market 1501 evaluation protocol uses fixed 3368 high-quality images as query. It is because we can easily draw high quality bounding box for query pedestrian.

In this sense, reis.datasets.market1501 did not follow the official evaluation protocol.

def register(subdir, pattern=re.compile(r'([-\d]+)_c(\d)')):
    fpaths = sorted(glob(osp.join(exdir, subdir, '*.jpg')))
    pids = set()
    for fpath in fpaths:
        fname = osp.basename(fpath)
        pid, cam = map(int, pattern.search(fname).groups())
        if pid == -1: continue  # junk images are just ignored
        assert 0 <= pid <= 1501  # pid == 0 means background
        assert 1 <= cam <= 6
        cam -= 1
        pids.add(pid)
        fname = ('{:08d}_{:02d}_{:04d}.jpg'
                 .format(pid, cam, len(identities[pid][cam])))
        identities[pid][cam].append(fname)
        shutil.copy(fpath, osp.join(images_dir, fname))
    return pids

trainval_pids = register('bounding_box_train')
gallery_pids = register('bounding_box_test')
query_pids = register('query')

This may explain why the demo code python examples/triplet_loss.py -d market1501 only has 67.9 mAP much lower than soa. For an visual case,

[zydou]

Please see issue #16. The code in issue #16 follows the offical splitting. And the author has explained the reason .

[luzai]

Indeed, there is little difference between splitting by query person IDs and splitting by query images.

Meanwhile, I am sorry for my mistake, the mAP of original paper is 69.14 using triplet+batch hard mining, and 81.07 after rerank. Open-reid has reached excellent 67.9!

[zydou]

The paper used ten crops for test-time augmentation while open-reid not. If applied test-time augmentation in open-reid, the results will be higher.

[Cysu]

@zydou Thank you very much for the explanation!

@luzai Thank you for pointing out the issue. Will try my best to fix it.