Reference Results

[ronshpindler]

Hi,

I am trying to understand the results meaning. I have compared one tracking ground truth file with itself and got the following: IDF1 IDP IDR Rcll Prcn GT MT PT ML FP FN IDs FM MOTA MOTP CVPR19-01 86.2% 75.8% 100.0% 100.0% 75.8% 74 74 0 0 6349 0 0 0 68.0% 0.000 OVERALL 86.2% 75.8% 100.0% 100.0% 75.8% 74 74 0 0 6349 0 0 0 68.0% 0.000

Shouldn't the MOTA and MOTP be 100%? Shouldn't the Prcn be 100%?

[cheind]

Interesting... For MOTP see here. That's fine.

For the others I have to speculate. How did you compute the distance between objects / hypotheses? The only way we get less than 100% for these two metrics is that we have false positives, ie. hypothesis that couldn't be matched to any object id. This happens for example if the distance between such pairs is NaN.

[ronshpindler]

Thanks Cristoph for the fast replay.

I got the results by simply using the eval_motchallenge.py. I have tried to run the evaluation tool on very small ground truth sample compared to its duplication. I have started with the first 10 objects of sample one and removed one object at the time. As you can see at the table below when the 9th object is added the evaluation tool starts to miss a detection.... This is the problem and I dont know why IDF1 IDP IDR Rcll Prcn GT MT PT ML FP FN IDs FM MOTA MOTP 10 CVPR19-01 93.0% 87.0% 100.0% 100.0% 87.0% 9 9 0 0 429 0 0 0 85.0% 0.000 9 CVPR19-01 91.9% 85.0% 100.0% 100.0% 85.0% 8 8 0 0 429 0 0 0 82.4% 0.000 8 CVPR19-01 100.0% 100.0% 100.0% 100.0% 100.0% 8 8 0 0 0 0 0 0 100.0% 0.000 7 CVPR19-01 100.0% 100.0% 100.0% 100.0% 100.0% 7 7 0 0 0 0 0 0 100.0% 0.000 6 CVPR19-01 100.0% 100.0% 100.0% 100.0% 100.0% 6 6 0 0 0 0 0 0 100.0% 0.000

[cheind]

Could you share the file? If you look into the test test_motchallenge_files and change to

def test_motchallenge_files_regression():
    dnames = [
        'TUD-Campus',
        'TUD-Stadtmitte',
    ]

    def compute_motchallenge(dname):
        df_gt = mm.io.loadtxt(os.path.join(dname,'test.txt')) # <-- change here
        df_test = mm.io.loadtxt(os.path.join(dname,'test.txt')) 
        return mm.utils.compare_to_groundtruth(df_gt, df_test, 'iou', distth=0.5)

    accs = [compute_motchallenge(os.path.join(DATA_DIR, d)) for d in dnames]

    mh = mm.metrics.create()
    summary = mh.compute_many(accs, metrics=mm.metrics.motchallenge_metrics, names=dnames, generate_overall=True)

    print()
    print(mm.io.render_summary(summary, namemap=mm.io.motchallenge_metric_names, formatters=mh.formatters))
    summary = summary[mm.metrics.motchallenge_metrics[:15]]
    print(summary)

then run with pytest -k test_motchallenge_files_regression -s. The will match your expectations. I suspect it has something todo with your file.