Different HOTA for the same ground-truth (DPM / FRCNN / SDP) of MOT17

[sieumap43]

Hi. Thanks for the great paper and repo. One phenomenon I see when running your script in the README.md is that the HOTA score of DPM, FRCNN and SDP are different, even though they should be the same. Their ground-truths gt.txt files also look the same. Can you have a look?

Command

    scripts/run_mot_challenge.py \
    --USE_PARALLEL True \
    --METRICS CLEAR HOTA Identity\
    --TRACKERS_TO_EVAL MPNTrack \

Output

Eval Config:
USE_PARALLEL         : True                          
NUM_PARALLEL_CORES   : 8                             
BREAK_ON_ERROR       : True                          
RETURN_ON_ERROR      : False                         
LOG_ON_ERROR         : /lhome/project-2/TrackEval/error_log.txt
PRINT_RESULTS        : True                          
PRINT_ONLY_COMBINED  : False                         
PRINT_CONFIG         : True                          
TIME_PROGRESS        : True                          
DISPLAY_LESS_PROGRESS : False                         
OUTPUT_SUMMARY       : True                          
OUTPUT_EMPTY_CLASSES : True                          
OUTPUT_DETAILED      : True                          
PLOT_CURVES          : True                          

MotChallenge2DBox Config:
PRINT_CONFIG         : True                          
GT_FOLDER            : /lhome/project-2/TrackEval/data/gt/mot_challenge/
TRACKERS_FOLDER      : /lhome/project-2/TrackEval/data/trackers/mot_challenge/
OUTPUT_FOLDER        : None                          
TRACKERS_TO_EVAL     : ['MPNTrack']                  
CLASSES_TO_EVAL      : ['pedestrian']                
BENCHMARK            : MOT17                         
SPLIT_TO_EVAL        : train                         
INPUT_AS_ZIP         : False                         
DO_PREPROC           : True                          
TRACKER_SUB_FOLDER   : data                          
OUTPUT_SUB_FOLDER    :                               
TRACKER_DISPLAY_NAMES : None                          
SEQMAP_FOLDER        : None                          
SEQMAP_FILE          : None                          
SEQ_INFO             : None                          
GT_LOC_FORMAT        : {gt_folder}/{seq}/gt/gt.txt   
SKIP_SPLIT_FOL       : False                         
GT_SET               :                               

CLEAR Config:
METRICS              : ['CLEAR', 'HOTA', 'Identity'] 
THRESHOLD            : 0.5                           
PRINT_CONFIG         : True                          

Identity Config:
METRICS              : ['CLEAR', 'HOTA', 'Identity'] 
THRESHOLD            : 0.5                           
PRINT_CONFIG         : True                          

Evaluating 1 tracker(s) on 21 sequence(s) for 1 class(es) on MotChallenge2DBox dataset using the following metrics: HOTA, CLEAR, Identity, Count

Evaluating MPNTrack

All sequences for MPNTrack finished in 4.44 seconds

HOTA: MPNTrack-pedestrian          HOTA      DetA      AssA      DetRe     DetPr     AssRe     AssPr     LocA      OWTA      HOTA(0)   LocA(0)   HOTALocA(0)
MOT17-02-DPM                       43.339    36.637    51.281    37.272    92.784    54.743    85.644    92.162    43.719    47.055    90.76     42.707    
MOT17-02-FRCNN                     48.721    43.967    54.013    44.99     92.076    58.298    82.601    91.715    49.295    53.28     89.991    47.947    
MOT17-02-SDP                       49.915    50.198    49.687    51.849    90.827    55.347    75.876    91.222    50.754    54.955    88.98     48.899    
MOT17-04-DPM                       66.112    59.231    73.857    61.008    92.566    76.027    93.578    91.892    67.126    72.457    90.393    65.496    
MOT17-04-FRCNN                     66.099    58.446    74.844    60.339    91.99     78.365    91.744    91.529    67.2      72.815    89.861    65.433    
MOT17-04-SDP                       70.858    67.158    74.944    70.108    90.92     78.746    91.218    90.971    72.478    79.12     88.864    70.309    
MOT17-05-DPM                       51.975    50.855    53.181    54.902    81.546    65.662    73.061    86.978    54.031    62.479    82.767    51.712    
MOT17-05-FRCNN                     53.255    51.028    55.627    55.104    81.183    69.218    71.153    86.455    55.362    65.858    81.921    53.951    
MOT17-05-SDP                       56.424    56.088    56.852    61.257    81.047    67.046    76.292    86.816    59.012    68.802    82.528    56.781    
MOT17-09-DPM                       64.741    68.035    61.624    70.848    90.71     71.732    78.184    90.913    66.064    72.545    89.321    64.798    
MOT17-09-FRCNN                     61.439    64.786    58.284    68.301    89.122    67.28     78.813    90.902    63.085    69.227    88.769    61.452    
MOT17-09-SDP                       65.392    68.447    62.491    71.484    90.373    72.88     78.839    90.873    66.823    73.473    89.27     65.59     
MOT17-10-DPM                       57.24     55.221    59.355    58.207    84.816    63.221    82.73     87.142    58.767    68.704    83.823    57.59     
MOT17-10-FRCNN                     60.968    62.692    59.334    66.897    83.68     63.295    83.129    86.484    62.994    74.458    83.032    61.824    
MOT17-10-SDP                       60.724    64.344    57.367    69.786    82.102    62.925    79.46     86.281    63.266    74.637    82.482    61.562    
MOT17-11-DPM                       63.56     61.162    66.075    62.92     92.971    70.333    89.555    92.404    64.476    69.014    91.205    62.944    
MOT17-11-FRCNN                     67.251    65.441    69.137    67.689    92.273    72.122    91.542    92.029    68.406    73.779    90.711    66.926    
MOT17-11-SDP                       69.269    70.202    68.366    73.071    91.64     71.794    90.542    91.737    70.671    76.754    90.143    69.188    
MOT17-13-DPM                       50.727    46.093    55.892    47.844    87.18     65.19     77.422    88.849    51.709    57.63     86.571    49.891    
MOT17-13-FRCNN                     62.309    63.483    61.267    68.412    83.925    71.553    75.187    88.061    64.739    71.919    85.638    61.59     
MOT17-13-SDP                       61.247    59.6      63.074    63.531    84.26     76.28     73.102    87.492    63.299    71.439    84.887    60.643    
COMBINED                           61.904    57.989    66.24     60.614    89.223    71.544    86.238    90.253    63.354    69.668    88.006    61.312    

CLEAR: MPNTrack-pedestrian         MOTA      MOTP      MODA      CLR_Re    CLR_Pr    MTR       PTR       MLR       sMOTA     CLR_TP    CLR_FN    CLR_FP    IDSW      MT        PT        ML        Frag      
MOT17-02-DPM                       39.04     91.754    39.137    39.653    98.714    17.742    35.484    46.774    35.77     7368      11213     96        18        11        22        29        19        
MOT17-02-FRCNN                     47.296    91.158    47.43     48.146    98.535    24.194    46.774    29.032    43.039    8946      9635      133       25        15        29        18        25        
MOT17-02-SDP                       53.662    90.655    53.888    55.487    97.2      27.419    50        22.581    48.477    10310     8271      297       42        17        31        14        45        
MOT17-04-DPM                       65.656    90.874    65.685    65.797    99.831    34.94     36.145    28.916    59.651    31291     16266     53        14        29        30        24        18        
MOT17-04-FRCNN                     65.288    90.372    65.303    65.448    99.779    38.554    34.94     26.506    58.987    31125     16432     69        7         32        29        22        13        
MOT17-04-SDP                       76.205    89.412    76.235    76.672    99.433    54.217    26.506    19.277    68.087    36463     11094     208       14        45        22        16        28        
MOT17-05-DPM                       55.906    85.711    56.166    61.746    91.711    24.06     48.872    27.068    47.083    4271      2646      386       18        32        65        36        29        
MOT17-05-FRCNN                     56.6      85.133    56.889    62.383    91.906    31.579    42.857    25.564    47.325    4315      2602      380       20        42        57        34        33        
MOT17-05-SDP                       62.137    85.628    62.455    69.018    91.316    36.842    45.113    18.045    52.217    4774      2143      454       22        49        60        24        38        
MOT17-09-DPM                       74.742    89.968    74.836    76.469    97.908    53.846    38.462    7.6923    67.07     4072      1253      87        5         14        10        2         6         
MOT17-09-FRCNN                     70.16     90.088    70.329    73.484    95.883    53.846    38.462    7.6923    62.876    3913      1412      168       9         14        10        2         8         
MOT17-09-SDP                       75.117    89.982    75.192    77.146    97.531    57.692    38.462    3.8462    67.389    4108      1217      104       4         15        10        1         6         
MOT17-10-DPM                       62.349    85.934    62.567    65.597    95.585    42.105    42.105    15.789    53.122    8422      4417      389       28        24        24        9         68        
MOT17-10-FRCNN                     72.155    85.026    72.529    76.236    95.362    61.404    33.333    5.2632    60.74     9788      3051      476       48        35        19        3         111       
MOT17-10-SDP                       73.799    84.853    74.359    79.679    93.741    68.421    28.07     3.5088    61.73     10230     2609      683       72        39        16        2         130       
MOT17-11-DPM                       64.985    91.968    65.091    66.384    98.09     24        48        28        59.653    6264      3172      122       10        18        36        21        16        
MOT17-11-FRCNN                     70.157    91.481    70.263    71.81     97.891    40        36        24        64.039    6776      2660      146       10        30        27        18        17        
MOT17-11-SDP                       75.318    91.221    75.456    77.596    97.315    48        34.667    17.333    68.505    7322      2114      202       13        36        26        13        24        
MOT17-13-DPM                       51.048    87.713    51.254    53.066    96.697    27.273    40        32.727    44.528    6178      5464      211       24        30        44        36        33        
MOT17-13-FRCNN                     70.718    86.805    71.053    76.284    93.583    58.182    31.818    10        60.652    8881      2761      609       39        64        35        11        73        
MOT17-13-SDP                       67.463    86.178    67.806    71.603    94.965    52.727    24.545    22.727    57.566    8336      3306      442       40        58        27        25        74        
COMBINED                           64.399    89.143    64.543    66.239    97.503    39.621    38.4      21.978    57.208    223153    113738    5715      482       649       629       360       814       

Identity: MPNTrack-pedestrian      IDF1      IDR       IDP       IDTP      IDFN      IDFP      
MOT17-02-DPM                       48.608    34.067    84.807    6330      12251     1134      
MOT17-02-FRCNN                     54.888    40.854    83.611    7591      10990     1488      
MOT17-02-SDP                       55.304    43.437    76.091    8071      10510     2536      
MOT17-04-DPM                       75.348    62.504    94.835    29725     17832     1619      
MOT17-04-FRCNN                     75.57     62.569    95.39     29756     17801     1438      
MOT17-04-SDP                       82.094    72.698    94.279    34573     12984     2098      
MOT17-05-DPM                       63.936    53.491    79.45     3700      3217      957       
MOT17-05-FRCNN                     63.228    53.072    78.19     3671      3246      1024      
MOT17-05-SDP                       68.588    60.214    79.667    4165      2752      1063      
MOT17-09-DPM                       77.33     68.864    88.17     3667      1658      492       
MOT17-09-FRCNN                     72.698    64.207    83.778    3419      1906      662       
MOT17-09-SDP                       77.488    69.39     87.726    3695      1630      517       
MOT17-10-DPM                       68.665    57.894    84.36     7433      5406      1378      
MOT17-10-FRCNN                     73.895    66.485    83.164    8536      4303      1728      
MOT17-10-SDP                       72.364    66.937    78.75     8594      4245      2319      
MOT17-11-DPM                       70.674    59.252    87.551    5591      3845      795       
MOT17-11-FRCNN                     76.513    66.32     90.407    6258      3178      664       
MOT17-11-SDP                       76.568    68.811    86.297    6493      2943      1031      
MOT17-13-DPM                       58.422    45.241    82.439    5267      6375      1122      
MOT17-13-FRCNN                     73.557    66.758    81.897    7772      3870      1718      
MOT17-13-SDP                       69.745    61.166    81.123    7121      4521      1657      
COMBINED                           71.206    59.79     88.011    201428    135463    27440     

Count: MPNTrack-pedestrian         Dets      GT_Dets   IDs       GT_IDs    
MOT17-02-DPM                       7464      18581     47        62        
MOT17-02-FRCNN                     9079      18581     54        62        
MOT17-02-SDP                       10607     18581     63        62        
MOT17-04-DPM                       31344     47557     72        83        
MOT17-04-FRCNN                     31194     47557     67        83        
MOT17-04-SDP                       36671     47557     78        83        
MOT17-05-DPM                       4657      6917      76        133       
MOT17-05-FRCNN                     4695      6917      85        133       
MOT17-05-SDP                       5228      6917      96        133       
MOT17-09-DPM                       4159      5325      22        26        
MOT17-09-FRCNN                     4081      5325      23        26        
MOT17-09-SDP                       4212      5325      21        26        
MOT17-10-DPM                       8811      12839     58        57        
MOT17-10-FRCNN                     10264     12839     74        57        
MOT17-10-SDP                       10913     12839     76        57        
MOT17-11-DPM                       6386      9436      56        75        
MOT17-11-FRCNN                     6922      9436      61        75        
MOT17-11-SDP                       7524      9436      67        75        
MOT17-13-DPM                       6389      11642     73        110       
MOT17-13-FRCNN                     9490      11642     90        110       
MOT17-13-SDP                       8778      11642     85        110       
COMBINED                           228868    336891    1344      1638      

[sieumap43]

Ah. The prediction files are different. For example MOT17-02-FRCNN.txt and MOT17-02-SDP.txt are different. Closing this now.

[ratnesh1729]

What is HOTA(0)?