Inconsistent Results on BDD100K Tracking Validation Set

[taheranjary]

Hi there.

I ran the pre-trained BDD100K model on the tracking validation set and the resulting MOTA IDF1 scores are lower than what QDTrack claim: MOTA: 54.5, IDF1: 66.7 vs your MOTA: 63.5, IDF1 71.5.

Kindly verify if this is the case for you or if there are any missing settings.

I followed the instructions and ran this command: sh ./tools/dist_test.sh ./configs/bdd100k/qdtrack-frcnn_r50_fpn_12e_bdd100k.py ./ckpts/mmdet/qdtrack_frcnn_r50_fpn_12e_bdd100k_13328aed.pth 2 --out exp.pkl --eval track

[joemathai]

Hi, I am facing the same issue as @taheranjary in reproducing the results.

@OceanPang Can you please clarify this?

[thomasehuang]

The discrepancy is due to the index of the categories being changed. The ground truth annotation file was probably updated such that the categories are now not aligned. I was able to reproduce the results after mapping the old categories: CLASSES = ('pedestrian', 'rider', 'car', 'bus', 'truck', 'bicycle', 'motorcycle', 'train') to the new categories: CLASSES = ('pedestrian', 'rider', 'car', 'truck', 'bus', 'train', 'motorcycle', 'bicycle') Notice the order of bus and truck as well as bicycle and train are changed.

Reproduced results for reference: Test set

            MOTA  MOTP  IDF1     FP      FN   IDSw     MT    PT    ML     FM
----------------------------------------------------------------------------
pedestrian  49.3  78.0  59.1  15232   35594   2396   2243  2155  1169   4460
rider       37.7  76.5  55.0    522    3597     30     42   107   116    211
car         71.4  84.6  76.7  52133  119959   7981  14688  6365  2828  12414
truck       32.9  84.9  53.6   6109   26818    270    227   545   513   1477
bus         31.4  85.6  55.4   3384    6880     80     82   122   139    555
train        2.8  59.9  16.7     35     454      0      0     1    10      8
motorcycle  28.4  77.5  50.4    446    2274     15     23    90    95     70
bicycle     31.6  75.5  51.8    998    4485     50    101   189   191    211
----------------------------------------------------------------------------
human       48.6  77.9  58.9  15754   39191   2426   2285  2262  1285   4671
vehicle     67.7  84.7  74.8  61661  154111   8331  14997  7033  3490  14454
bike        30.6  76.1  51.4   1444    6759     65    124   279   286    281
----------------------------------------------------------------------------
AVERAGE     35.7  77.8  52.3  78859  200061  10822  17406  9574  5061  19406
OVERALL     64.5  83.8  72.4  78859  200061  10822  17406  9574  5061  19406

Val set

            MOTA  MOTP  IDF1     FP      FN  IDSw    MT    PT    ML     FM
--------------------------------------------------------------------------
pedestrian  49.3  78.4  59.9   8743   18795  1419  1460  1360   670   2471
rider       35.0  77.5  51.5    144    1492    15    11    55    65     85
car         69.8  84.6  75.0  27255   71443  4519  7451  3786  1926   7058
truck       39.2  85.4  58.2   2934   13593   144   132   317   276    855
bus         40.8  86.2  62.3   1588    3738    52    57    90    46    336
train        0.0   nan   0.0      0     308     0     0     0     6      0
motorcycle  28.8  76.9  56.0    170     467     5     7    18    19     28
bicycle     30.0  76.2  50.1    521    2336    39    51    98    95    125
--------------------------------------------------------------------------
human       48.7  78.4  59.6   8887   20287  1434  1471  1415   735   2556
vehicle     66.8  84.7  73.6  31777   89082  4715  7640  4193  2254   8249
bike        29.8  76.3  51.2    691    2803    44    58   116   114    153
--------------------------------------------------------------------------
AVERAGE     36.6  70.7  51.6  41355  112172  6193  9169  5724  3103  10958
OVERALL     63.9  83.9  71.5  41355  112172  6193  9169  5724  3103  10958

[long2double]

Hi, Modify the CLASSES of BDDVideodataset to new categories CLASSES = ('pedestrian', 'rider', 'car', 'truck', 'bus', 'train', 'motorcycle', 'bicycle'), but the result is still the same as the old one? Please explain in detail how to operate or update the new code?

[joemathai]

I was able to reproduce the results in the paper (eval set) by remapping the category_ids in coco-formatted label files as explained in the previous post. You might be facing issues because the pre-trained model was trained with different category_id mapping and might not match the new category_id mapping in BDDVideodataset.

@thomasehuang how do I get the labels for the test set of BDD 100k MOT? I can't seem to find it in the official website.

The discrepancy is due to the index of the categories being changed. The ground truth annotation file was probably updated such that the categories are now not aligned. I was able to reproduce the results after mapping the old categories: CLASSES = ('pedestrian', 'rider', 'car', 'bus', 'truck', 'bicycle', 'motorcycle', 'train') to the new categories: CLASSES = ('pedestrian', 'rider', 'car', 'truck', 'bus', 'train', 'motorcycle', 'bicycle') Notice the order of bus and truck as well as bicycle and train are changed.

Reproduced results for reference: Test set

            MOTA  MOTP  IDF1     FP      FN   IDSw     MT    PT    ML     FM
----------------------------------------------------------------------------
pedestrian  49.3  78.0  59.1  15232   35594   2396   2243  2155  1169   4460
rider       37.7  76.5  55.0    522    3597     30     42   107   116    211
car         71.4  84.6  76.7  52133  119959   7981  14688  6365  2828  12414
truck       32.9  84.9  53.6   6109   26818    270    227   545   513   1477
bus         31.4  85.6  55.4   3384    6880     80     82   122   139    555
train        2.8  59.9  16.7     35     454      0      0     1    10      8
motorcycle  28.4  77.5  50.4    446    2274     15     23    90    95     70
bicycle     31.6  75.5  51.8    998    4485     50    101   189   191    211
----------------------------------------------------------------------------
human       48.6  77.9  58.9  15754   39191   2426   2285  2262  1285   4671
vehicle     67.7  84.7  74.8  61661  154111   8331  14997  7033  3490  14454
bike        30.6  76.1  51.4   1444    6759     65    124   279   286    281
----------------------------------------------------------------------------
AVERAGE     35.7  77.8  52.3  78859  200061  10822  17406  9574  5061  19406
OVERALL     64.5  83.8  72.4  78859  200061  10822  17406  9574  5061  19406

Val set

            MOTA  MOTP  IDF1     FP      FN  IDSw    MT    PT    ML     FM
--------------------------------------------------------------------------
pedestrian  49.3  78.4  59.9   8743   18795  1419  1460  1360   670   2471
rider       35.0  77.5  51.5    144    1492    15    11    55    65     85
car         69.8  84.6  75.0  27255   71443  4519  7451  3786  1926   7058
truck       39.2  85.4  58.2   2934   13593   144   132   317   276    855
bus         40.8  86.2  62.3   1588    3738    52    57    90    46    336
train        0.0   nan   0.0      0     308     0     0     0     6      0
motorcycle  28.8  76.9  56.0    170     467     5     7    18    19     28
bicycle     30.0  76.2  50.1    521    2336    39    51    98    95    125
--------------------------------------------------------------------------
human       48.7  78.4  59.6   8887   20287  1434  1471  1415   735   2556
vehicle     66.8  84.7  73.6  31777   89082  4715  7640  4193  2254   8249
bike        29.8  76.3  51.2    691    2803    44    58   116   114    153
--------------------------------------------------------------------------
AVERAGE     36.6  70.7  51.6  41355  112172  6193  9169  5724  3103  10958
OVERALL     63.9  83.9  71.5  41355  112172  6193  9169  5724  3103  10958

[long2double]

@joemathai joemathai, Please explain in detail how to operate? Thanks!!

[joemathai]

A simple python script like this can be used to remap the box_track_train_cocofmt.json and box_track_val_cocofmt.json to the category_ids described earlier.

[thomasehuang]

@joemathai Unfortunately the test labels are not publicly available.

[fyu]

@joemathai You can get evaluation results on the test set here https://eval.ai/web/challenges/challenge-page/1259/overview