test speed and trained result not the same as given

[Dawn-JN]

Hi, I have two problems.

1. About test speed,I tried to run fairmot_dla34.pth on a single 2080ti, the speed of 2DMOT15 is only 18 fps.No files are modified except for the path to datasets, and I download the code from your github,here is the result,I didn't save images or videos when running

   root@I4f3943c4d00601cef:/hy-tmp/FairMOT/src# python track.py mot --load_model ../models/fairmot_dla34.pth --conf_thres 0.6
   Fix size testing.
   training chunk_sizes: [6, 6]
   The output will be saved to  /hy-tmp/FairMOT/src/lib/../../exp/mot/default
   heads {'hm': 1, 'wh': 4, 'id': 128, 'reg': 2}
   2021-07-07 10:44:26 [INFO]: start seq: KITTI-13
   Creating model...
   loaded ../models/fairmot_dla34.pth, epoch 30
   ...............
   2021-07-07 10:47:04 [INFO]: Time elapsed: 114.63 seconds, FPS: 17.93
               IDF1   IDP   IDR  Rcll  Prcn  GT MT PT ML  FP   FN IDs   FM  MOTA  MOTP IDt IDa IDm
   KITTI-13       67.5% 71.3% 64.1% 67.3% 74.8%  42 10 24  8 197  285  17   28 42.7% 0.231   5  12   0
   KITTI-17       72.7% 83.7% 64.3% 70.1% 91.2%   9  2  7  0  52  231  11   23 62.0% 0.198   6   4   0
   ADL-Rundle-6   74.4% 80.2% 69.4% 79.3% 91.7%  24 16  8  0 360 1037  11   78 71.9% 0.195   9   5   3
   PETS09-S2L1    87.4% 89.0% 85.8% 93.3% 96.7%  19 18  1  0 150  313   9   90 89.8% 0.250   2   7   1
   TUD-Campus     71.9% 80.1% 65.2% 79.4% 97.6%   8  6  2  0   7   74   3   12 76.6% 0.198   3   0   0
   TUD-Stadtmitte 81.6% 92.1% 73.3% 79.2% 99.5%  10  8  2  0   5  241   4    9 78.4% 0.248   2   3   1
   OVERALL        79.3% 84.2% 74.9% 83.0% 93.2% 112 60 44  8 771 2181  55  240 76.5% 0.224  27  31   5

2. About trained result, I have trained dla34 with

   sh experiments/crowdhuman_dla34.sh
   sh experiments/mix_ft_ch_dla34.sh

   And the ctdet_coco_dla_2x.pth has been loaded ,because there are some warning about the weight is not fully loaded. I think it's normal,because FairMOT's heads are different from CenterNet,but I'm not sure.Then, I test with 2DMOT15, got 75.8 MOTA and 77.6 IDF1(76.5MOTA and 79.3IDF1 in README). I also use MOT17 Seg data (MOT17-02,MOT17-05,MOT17-09,MOT17-11), got 65.4 MOTA and 62.9 IDF1 (70.2MOTA and 64IDF1 in paper table 8).I didn't use CH to pretrain for MOT17-Seg, should I?

Could you help me find out where is wrong?Appreciate for your answer.

[Dawn-JN]

I found that the author said "I do not train the parameter of the 'id_classifier'. You can uncomment this line: FairMOT/src/lib/trains/base_trainer.py Line 30 in 1851158", in other issuse, but the code is not comment, so I comment this:

self.optimizer.add_param_group({'params': self.loss.parameters()}) 

and use MOT17 Seg to train, got

root@I4f3943c4d00601cef:/hy-tmp/FairMOT/src# python track.py mot --load_model ../exp/mot/mot17_seg_dla34_unid/model_last.pth --conf_thres 0.6
Fix size testing.
training chunk_sizes: [6, 6]
The output will be saved to  /hy-tmp/FairMOT/src/lib/../../exp/mot/default
heads {'hm': 1, 'wh': 4, 'id': 128, 'reg': 2}
2021-07-07 18:38:23 [INFO]: start seq: KITTI-13
Creating model...
loaded ../exp/mot/mot17_seg_dla34_unid/model_last.pth, epoch 30
............
2021-07-07 18:41:03 [INFO]: Time elapsed: 116.50 seconds, FPS: 17.64
                IDF1   IDP   IDR  Rcll  Prcn  GT MT PT ML   FP   FN IDs   FM  MOTA  MOTP IDt IDa IDm
KITTI-13       42.9% 38.3% 48.7% 62.6% 49.2%  42 12 18 12  591  342  57   52 -8.2% 0.235  13  46   3
KITTI-17       74.5% 81.9% 68.3% 76.7% 92.0%   9  4  5  0   52  182   9   25 68.9% 0.234   2   5   0
ADL-Rundle-6   66.1% 73.2% 60.3% 76.4% 92.7%  24 15  8  1  302 1182  31   90 69.8% 0.191  18  15   3
PETS09-S2L1    75.3% 78.6% 72.2% 87.9% 95.8%  19 16  3  0  181  561  24  164 83.5% 0.279   4  16   1
TUD-Campus     72.6% 75.6% 69.9% 87.7% 94.9%   8  6  2  0   17   44   7    9 81.1% 0.253   3   2   1
TUD-Stadtmitte 80.9% 89.6% 73.7% 79.4% 96.5%  10  6  4  0   33  238   7   15 76.0% 0.338   3   4   1
OVERALL        69.4% 73.6% 65.7% 80.2% 89.8% 112 59 40 13 1176 2549 135  355 70.0% 0.246  43  88   9

The results have improved,but IDs = 135 (IDs=96 in paper),and fps is still 18

[zengjie617789]

hello, after reading you problem, i have two doubts:

1. How can see it importved? The results on KITTI-13 is much worse than before though it indeed detect more objects which are most incorrect;
2. What mean the author refered "I do not train the parameter of the 'id_classifier'? How does model learn the abstract features if we didnot train the classifier? As far as i am concerned, it means the whole model trained the params except the part of the combined loss;
3. Finaly, i really cannot reach the fps speed when test the model on MOT16 datasets. I will be appreciate if you can offer help. Thank you in advance.

[Dawn-JN]

hello, after reading you problem, i have two doubts:

1. How can see it importved? The results on KITTI-13 is much worse than before though it indeed detect more objects which are most incorrect;
2. What mean the author refered "I do not train the parameter of the 'id_classifier'? How does model learn the abstract features if we didnot train the classifier? As far as i am concerned, it means the whole model trained the params except the part of the combined loss;
3. Finaly, i really cannot reach the fps speed when test the model on MOT16 datasets. I will be appreciate if you can offer help. Thank you in advance.

hello,I said the results have improved,just beacuse overall results are close to the paper,I think there are still some problems,but I don't know where.Did you get the same result as in the paper ? I will be appreciate if you can help.

And about the author's words, you can search Issue "About the log file",the author said in his reply, "In fact, I do not train the parameter of the 'id_classifier', so the CE loss is high. I find that using this way can get higher IDF1 score. You can also add the parameters of the 'id_classifier' to the optimizer and the CE loss can drop to < 1. Training the 'id_classifier' can get lower id switches but still lower IDF1 on the validation set, which is a little bit strange. I'm not sure whether training the parameters of 'id_classifier' can get better results on the test set of MOT challenge." You can search and check it.

Finally,you said you cannot reach the fps speed when test the model on MOT16 datasets.Did you can reach the fps speed when test the model on other datasets?I test on MOT15,and the fps is only 18 (2080ti),I don't know why. I will be appreciate if you can offer help.

[zengjie617789]

@Dawn-JN , hi, here is the results of fairmot.model tested on MOT16. I trained it by myself and got the similar results. I also used YOLOv5m as backbone for training, here is the rough results i got: As for the problem of "id_classifier", i will look into in later.

[Dawn-JN]

@Dawn-JN , hi, here is the results of fairmot.model tested on MOT16. I trained it by myself and got the similar results. I also used YOLOv5m as backbone for training, here is the rough results i got: As for the problem of "id_classifier", i will look into in later.

Thanks a lot,and how about the fps?

[Dawn-JN]

I trained MOT17seg again with 4 k80 gpu(rental 2 2080ti before),and got the similar result. As for the fps,when testing MOT17-04-SDP,the fps of GTX1650(laptop) is 6.7，but the fps of rental 2080ti is less than 10.I think it is not normal,could it be the gpu’s problem?

[zengjie617789]

hello @Dawn-JN, i record the time at line blow which cost much： https://github.com/ifzhang/FairMOT/blob/815d6585344826e0346a01efd57de45498cfe52b/src/lib/tracker/multitracker.py#L257 I add torch.cuda.synchronize() at line 183, but it seems make no difference. https://github.com/ifzhang/FairMOT/blob/815d6585344826e0346a01efd57de45498cfe52b/src/lib/tracker/multitracker.py#L183 you can test by youself and it will be glad to see your shared results.

[Dawn-JN]

hi, @zengjie617789 ,I changed another 2080ti GPU on the server rental platform.The fps testing on MOT15 trainsets is 21(less than 22),on MOT17 testsets is 13.15.The fps on MOT15 is better than before.So I think it may because the GPU.

And I have tested the method you said,but it makes no difference.

I will try to test on other GPUs, test the time to see if it's useful

[zengjie617789]

@Dawn-JN ,hello, what do you mean that "I have tested the method you said,but it makes no difference"? Is it you recored the time at that line and it costed a trivial time?

[Dawn-JN]

@Dawn-JN ,hello, what do you mean that "I have tested the method you said,but it makes no difference"? Is it you recored the time at that line and it costed a trivial time?

oh,sorry,I thought add torch.cuda.synchronize() could make the time calculation more accurate,as well as fps.The time of that line is a fixed value,I also recored the time of updata function,

    def update(self, im_blob, img0):
        torch.cuda.synchronize()
        start = time.time()
....
        test = time.time()
        id_feature = id_feature.cpu().numpy()
        test1 = time.time()
        print('id_feature time', test1 - test)
....
        torch.cuda.synchronize()
        end = time.time()
        print('track time:', end - start)
        return output_stracks

This is the result of my laptop(1650),

2021-07-09 10:02:41 [INFO]: start seq: KITTI-13
Creating model...
loaded ..\models\fairmot_dla34.pth, epoch 30
2021-07-09 10:02:43 [INFO]: Processing frame 0 (100000.00 fps)
id_feature time 0.0780026912689209
track time: 1.159336805343628
id_feature time 0.12302565574645996
track time: 0.15703487396240234
id_feature time 0.10901999473571777
track time: 0.13609862327575684
id_feature time 0.10502266883850098
track time: 0.13502168655395508
id_feature time 0.10703897476196289
track time: 0.13502931594848633
......

and 2080ti

root@I5052057e900401e7c:/hy-tmp/FairMOT/src# python track.py mot --load_model ../models/fairmot_dla34.pth --conf_thres 0.6
Fix size testing.
training chunk_sizes: [6, 6]
The output will be saved to  /hy-tmp/FairMOT/src/lib/../../exp/mot/default
heads {'hm': 1, 'wh': 4, 'id': 128, 'reg': 2}
2021-07-09 10:13:15 [INFO]: start seq: KITTI-13
Creating model...
loaded ../models/fairmot_dla34.pth, epoch 30
2021-07-09 10:13:18 [INFO]: Processing frame 0 (100000.00 fps)
id_feature time 0.0063130855560302734
track time: 0.06557464599609375
id_feature time 0.014795780181884766
track time: 0.05182838439941406
id_feature time 0.013593912124633789
track time: 0.05222797393798828
id_feature time 0.013605356216430664
track time: 0.05173945426940918
id_feature time 0.00643610954284668
track time: 0.0437009334564209
id_feature time 0.00637364387512207
track time: 0.042917728424072266
id_feature time 0.0064487457275390625
track time: 0.04266524314880371
id_feature time 0.0060503482818603516
track time: 0.04276013374328613
id_feature time 0.006331920623779297
track time: 0.04250931739807129
id_feature time 0.006075382232666016
track time: 0.0426332950592041
....

You can tell the diffence easily,It spent more time elsewhere on 2080ti,I will try to find where.Could you share your run time of these two place?

[MikeMegamind]

Hello, I got the same problem. I test on a 3070ti gpu but only got nearly 20 FPS. I don't know how to improve the tracking speed.

[Dawn-JN]

Hello, I got the same problem. I test on a 3070ti gpu but only got nearly 20 FPS. I don't know how to improve the tracking speed.

Sorry, I didn't solve the problem.The gpus I used to test are all rented, and I only found that the CPU processing time is too long.But I don't have another computer (better CPU) to test with.You can check your CPU and GPU processing speed and try to improve. (I finally tested the speed with my laptop, because the CPU speed on the laptop is relatively normal)

[MikeMegamind]

All right... Thank you all the same.

[cgg99]

@Dawn-JN Why do I always get 0% and NaN for each indicator when I test on MOT17 orMOT15？