garrickbrazil
commented
5 years ago It's interesting that your performance is so high (more so than ours main experiment). Unfortunately, we cannot guarantee the same performance as our experiments. We generally observe that the "car" experiment is the most stable, which seems to at least correspond to the high-performing final results of your experiments (17.45% on moderate 3D Car).
The largest performance improvement of our paper comes from the formulation of M3D-RPN and a modest gain comes from using depth-aware layers at the end of the network. As described in Sec 4.2, the performance increase is ~1.98% on 3D car (as opposed to 6%). For clarity, the numbers you are reporting correspond to Row 2 and Row 7 of Tab. 5.
When comparing our performance gains against prior art and ablations it goes from 5.69% [prior art] -> 15.08% [base] -> 17.06% [with depth-aware]. It seems that your warmup experiment went so well that the improvement with depth-aware enabled is marginalised.
liulj13
When I trained the model with and without depth-aware convolution on val1, I found that there is no significant difference between them. The results are what we trained without depth-aware:
test_iter 50000 2d car --> easy: 0.9041, mod: 0.8394, hard: 0.6780 test_iter 50000 gr car --> easy: 0.2736, mod: 0.2127, hard: 0.1770 test_iter 50000 3d car --> easy: 0.2100, mod: 0.1702, hard: 0.1522 test_iter 50000 2d pedestrian --> easy: 0.6733, mod: 0.5882, hard: 0.5056 test_iter 50000 gr pedestrian --> easy: 0.0689, mod: 0.0677, hard: 0.0646 test_iter 50000 3d pedestrian --> easy: 0.0397, mod: 0.0385, hard: 0.0328 test_iter 50000 2d cyclist --> easy: 0.6383, mod: 0.4694, hard: 0.4062 test_iter 50000 gr cyclist --> easy: 0.0545, mod: 0.0563, hard: 0.0565 test_iter 50000 3d cyclist --> easy: 0.0432, mod: 0.0555, hard: 0.0557
And the results we trained with depth-aware is:
test_iter 50000 2d car --> easy: 0.9210, mod: 0.8443, hard: 0.6824 test_iter 50000 gr car --> easy: 0.2716, mod: 0.2195, hard: 0.1789 test_iter 50000 3d car --> easy: 0.2136, mod: 0.1745, hard: 0.1554 test_iter 50000 2d pedestrian --> easy: 0.6637, mod: 0.5836, hard: 0.5011 test_iter 50000 gr pedestrian --> easy: 0.0583, mod: 0.0561, hard: 0.0492 test_iter 50000 3d pedestrian --> easy: 0.0565, mod: 0.0529, hard: 0.0439 test_iter 50000 2d cyclist --> easy: 0.7268, mod: 0.4851, hard: 0.4810 test_iter 50000 gr cyclist --> easy: 0.0712, mod: 0.0584, hard: 0.0561 test_iter 50000 3d cyclist --> easy: 0.0594, mod: 0.0549, hard: 0.0551
As the Tab. 5 in the paper, there should be a significant improvement in 3D and BEV performance with depth-aware (about 6% and 8%) compared to that without depth-aware, but we do not find that (only about 0.4% and 0.7%). And we also notice that the performance of pedestrain and cyclist is also not as good, only 5% for pedestrain and cyclist, but in Tab. 3, it is about 10%.