search

zhaoweicai / mscnn

Caffe implementation of our multi-scale object detection framework
404 stars 211 forks source link

Test dataset accuracy #64

Open du0002in opened 7 years ago

du0002in commented 7 years ago

Hi, @zhaoweicai

May I check whether you have encountered the following problem on the test dataset accuracy? Do you have any suggestions to solve it?

I trained my own network using the KITTI train set and evaluate the accuracy using the val dataset. The way of splitting the two sets are exactly the same as what was described in your paper. I obtained 89% accuracy on the car detection (moderate group). But when I applied the same network on the test dataset and submitted the results to KITTI, the preview page showed only ~75% accuracy for moderate group. For the same network, I also tried to train it using both the train and val dataset and the accuracy on the test dataset was only improved to ~80%, which was still quite bad as compared to the 89% in the val dataset.

Thanks.

zhaoweicai commented 7 years ago

Hi @du0002in I agree that the train/val splitting on KITTI is a little bit tricky. However, the splitting I used is kind of reasonable. Although there are some degradation when I validate the model on testing set, I have never experienced \~15% degradation like you have. But degradation like 5\~8% could happen I think. It is hard for me to tell what your problems are, since I didn't validate my model on testing too much.

zxduan90 commented 6 years ago

Hi, @du0002in I have enconter the same problem as yours, have you solved the problem?