Classes Pedestrian and Cyclist much lower than paper

[Chen-Bo-Yang]

I uesd released weight on GitHub, KITTI DLA-34 and KITTI V2-99. And I tried to use both weight to do evaluation on validation set. I could get similer results on Car class, but on other classes are all limited to zero. I known the TTA and different evaluation sets are cause variance. But the variance should not be so much. Are there any missing settings that are causing this issue? Below table is my evaluate results:

Car AP R40@0.7		Paper	KITTI Submit	KITTI DLA	KITTI V2
BEV AP	Easy	30.98	32.35	31.65	40.70
Med	22.56	23.41	24.43	32.04
Hard	20.03	20.42	21.72	28.54
3D AP	Easy	23.22	23.19	22.56	30.38
Med	16.34	16.87	16.98	23.73
Hard	14.20	14.36	14.93	20.88

Pedestrian AP R40@0.7		Paper	KITTI Submit	KITTI DLA	KITTI V2
BEV AP	Easy	15.90	18.58	0.04	0.538
Med	10.85	12.51	0.03	0.056
Hard	8.05	10.65	0.01	0.028
3D AP	Easy	13.91	16.64	0.007	0.017
Med	9.30	11.04	0.008	0.018
Hard	8.05	9.38	0.009	0.019

Cyclist AP R40@0.7		Paper	KITTI Submit	KITTI DLA	KITTI V2
BEV AP	Easy	3.20	9.20	0.44	0.447
Med	1.99	5.69	0.27	0.229
Hard	1.79	5.20	0.28	0.258
3D AP	Easy	2.39	7.52	0.13	0.126
Med	1.52	4.79	0.12	0.123
Hard	1.31	4.22	0.11	0.111

And I have another question about published results, why it has differece between arxiv paper and kitti submit.

Thank you so much.

[sanmin0312]

Hello. Your results for the pedestrian class and cyclist class are based on IOU=0.7. However, KITTI evaluates pedestrian and cyclist based on IOU=0.5 (While the car class is evaluated on IOU=0.7) You can find the evaluation criterion in http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d

[Chen-Bo-Yang]

Very thank for your reply, it's my mistake.