How to eval on MOT16

[TaylorWillber]

     IDF1 IDP  IDR Rcll Prcn GT MT PT ML FP   FN IDs  FM MOTA MOTP IDt IDa IDm

MOT16-09 0.0% NaN 0.0% 0.0% NaN 25 0 0 25 0 5257 0 0 0.0% NaN 0 0 0 OVERALL 0.0% NaN 0.0% 0.0% NaN 25 0 0 25 0 5257 0 0 0.0% NaN 0 0 0 I write the code by yolov3_deepsort,that code can run the results.but this get this results.Do you run this code on MOT16 ?

[mikel-brostrom]

Hi! Sorry for my late reply. I have actually not run this on MOT16. I just made the MOT values available but never tried it due to lack of time. Do you have a solution for this??

[zengjie617789]

hello, i run that code to generate results to match MOT format. It seems that the code should change the format of xyxy to tlwh. After I recorrect it , the results i got is too bad. Here are results below: HOTA: MPNTrack-pedestrian HOTA DetA AssA DetRe DetPr AssRe AssPr LocA RHOTA HOTA(0) LocA(0) HOTALocA(0) MOT16-02 2.2651 1.002 5.9265 1.2815 4.0801 13.763 8.8246 63.03 2.5691 6.4246 21.604 1.388

[mikel-brostrom]

I see...

https://github.com/mikel-brostrom/Yolov5_DeepSort_Pytorch/blob/64c59e17fb570c3626c459f6a0216d8b46a074bd/deep_sort_pytorch/deep_sort/deep_sort.py#L26

Returns (x1, y1, x2, y2) and not (l, t, w, d), which is the MOT16 format needed according to:

https://motchallenge.net/instructions/

[zengjie617789]

I see...

https://github.com/mikel-brostrom/Yolov5_DeepSort_Pytorch/blob/64c59e17fb570c3626c459f6a0216d8b46a074bd/deep_sort_pytorch/deep_sort/deep_sort.py#L26

Returns (x1, y1, x2, y2) and not (t, l, w, d), which is the MOT16 format needed according to:

https://motchallenge.net/instructions/

Thank you for answering. Indeed, the function returned the results in form of xyxy. I used the bbox_rel function to change the data and put them into the mot challenge evaluation.

[zengjie617789]

OMG, I ignored the format the function returned is x_center, y_center, w, h. I will give a try tomorrow~ thank you !

[mikel-brostrom]

Let me know what MOT results you get! :nerd_face:

[mikel-brostrom]

I implemented a fix for this. You should be able to evaluate this solution on MOT now

[zengjie617789]

@mikel-brostrom hi, The reason that caused the results is the label format should be modified to "bbox_top, bbox_left, width, height"

f.write(('%g ' * 10 + '\n') % (frame_idx, identity, bbox_left, bbox_top, bbox_w, bbox_h, -1, -1, -1, -1)) # label formal to f.write(('%g ' * 10 + '\n') % (frame_idx, identity, bbox_top, bbox_left, bbox_w, bbox_h, -1, -1, -1, -1)) # label format

Here are results blow:

[zengjie617789]

using yolov5_crowdhuman weights results are blow hers:

[mikel-brostrom]

But these metrics are only for one of the MOT16 sequences right, nr04? Preliminary, on one sequence, compared to other tracker we have the following then:

Tracker	MOTA	IDF1	MT	ML	IDs	FPS
EAMTT	52.5	53.3	19.9%	34.9%	910	<5.5
SORTwHPD16	59.8	53.8	25.4%	22.7%	1423	<8.6
DeepSORT2	61.4	62.2	32.8%	18.2%	781	<6.4
RAR16wVGG	63.0	63.8	39.9%	22.1%	482	<1.4
VMaxx	62.6	49.2	32.7%	21.1%	1389	<3.9
TubeTK	64.0	59.4	33.5%	19.4%	1117	1.0
JDE	64.4	55.8	35.4%	20.0%	1544	18.5
TAP	64.8	73.5	38.5%	21.6%	571	<8.0
CNNMTT	65.2	62.2	32.4%	21.3%	946	<5.3
POI	66.1	65.1	34.0%	20.8%	805	<5.0
CTrackerV1	67.6	57.2	32.9%	23.1%	1897	6.8
FairMOT	74.9	72.8	44.7%	15.9%	1074	25.9
yolov5_crowdhuman+deep_sort	63.9	65.7	33%	15%	96	?

But MOT16-04 is one of the hardest in the dataset so the metric should be higher when evaluated on all sequences.

[zengjie617789]

yes, I will test it on the whole data later. Thank you very much. and i think you should correct the code mentioned above in case of others got a wrong results.

[mikel-brostrom]

Could you write instructions, step by step, for how to evaluate this? It would be nice to add that to the README. Feel free to create a pull request :smiley:

[mikel-brostrom]

I get the following train results using the MOT evaluation script here: https://github.com/JonathonLuiten/TrackEval which according to: https://github.com/dendorferpatrick/MOTChallengeEvalKit is 'the new official python (MOT) evaluation code'

Moreover, I use this crowd human model: https://github.com/deepakcrk/yolov5-crowdhuman

CLEAR: humancrowd_yolov5_deep_sort-pedestrianMOTA      MOTP      MODA      CLR_Re    CLR_Pr    MTR       PTR       MLR       sMOTA     CLR_TP    CLR_FN    CLR_FP    IDSW      MT        PT        ML        Frag      
MOT16-02                           34.363    76.061    34.969    42.186    85.392    16.667    42.593    40.741    24.265    7523      10310     1287      108       9         23        22        240       
MOT16-04                           60.09     75.95     60.298    74.176    84.239    42.169    40.964    16.867    42.25     35276     12281     6600      99        35        34        14        341       
MOT16-05                           54.488    76.493    55.603    72.998    80.756    44        47.2      8.8       37.328    4977      1841      1186      76        55        59        11        156       
MOT16-09                           60.167    81.933    61.138    78.258    82.05     56        36        8         46.029    4114      1143      900       51        14        9         2         74        
MOT16-10                           52.931    74.668    53.824    64.166    86.119    29.63     53.704    16.667    36.676    7904      4414      1274      110       16        29        9         333       
MOT16-11                           58.208    83.949    58.633    80.663    78.548    55.072    34.783    10.145    45.261    7400      1774      2021      39        38        24        7         82        
MOT16-13                           30.934    68.282    31.948    47.066    75.688    12.15     57.944    29.907    16.006    5389      6061      1731      116       13        62        32        330       
COMBINED                           51.614    76.444    52.156    65.741    82.874    34.816    46.422    18.762    36.128    72583     37824     14999     599       180       240       97        1556      

Identity: humancrowd_yolov5_deep_sort-pedestrianIDF1      IDR       IDP       IDTP      IDFN      IDFP      
MOT16-02                           32.556    24.32     49.228    4337      13496     4473      
MOT16-04                           64.234    60.397    68.591    28723     18834     13153     
MOT16-05                           41.353    39.366    43.55     2684      4134      3479      
MOT16-09                           51.757    50.561    53.012    2658      2599      2356      
MOT16-10                           56.792    49.553    66.507    6104      6214      3074      
MOT16-11                           50.605    51.286    49.942    4705      4469      4716      
MOT16-13                           36.931    29.948    48.16     3429      8021      3691      
COMBINED                           53.175    47.678    60.104    52640     57767     34942     

Count: humancrowd_yolov5_deep_sort-pedestrianDets      GT_Dets   IDs       GT_IDs    
MOT16-02                           8810      17833     57        54        
MOT16-04                           41876     47557     115       83        
MOT16-05                           6163      6818      43        125       
MOT16-09                           5014      5257      25        25        
MOT16-10                           9178      12318     52        54        
MOT16-11                           9421      9174      62        69        
MOT16-13                           7120      11450     59        107       
COMBINED                           87582     110407    413       517       

Timing analysis:
MotChallenge2DBox.get_raw_seq_data                                     1.5145 sec
MotChallenge2DBox.get_preprocessed_seq_data                            1.4160 sec
CLEAR.eval_sequence                                                    0.3576 sec
Identity.eval_sequence                                                 0.0680 sec
Count.eval_sequence                                                    0.0000 sec
eval_sequence                                                          3.3743 sec
Evaluator.evaluate                                                     3.3760 sec

[mikel-brostrom]

I see quite a big difference between our results for MOT16-04. Which evaluation script are you using @zengjie617789 ?

[mikel-brostrom]

However, I added instructions on how to evaluate to: https://github.com/mikel-brostrom/Yolov5_DeepSort_Pytorch/wiki/Evaluation At some point a would like to automate the process but right now I will leave it like that :smiley:

[zengjie617789]

hi, @mikel-brostrom , i used the same TrackEval package and the crowdhuman weigths as you . The only difference between us is i used the conf_thres 0.5 when i generate the track results.

[mikel-brostrom]

I acknowledged you, @zengjie617789 here for your contributions. Thank you so much for taking your time helping me out!