search

cxy1997 / 3D_adapt_auto_driving

Train in Germany, Test in The USA: Making 3D Object Detectors Generalize
https://arxiv.org/abs/2005.08139
130 stars 13 forks source link

nuScenes evaluation #8

Open kaancayli opened 3 years ago

kaancayli commented 3 years ago

Hi, I'm currently working on my bachelor thesis. In my work, I convert the nuScenes dataset into the KITTI format. Your work helped me a lot. I would like to thank you in advance for that.

I have a small problem. I'm using Frustum PointNet as a network architecture. After I convert the nuScenes dataset into the KITTI format, I train the network on this converted nuScenes dataset. Everything works fine but, when I want to evaluate the network performance, I encounter with some problems. I use KITTI benchmark for evaluation. The problem is, I get 3D AP scores close to 0, if I evaluate the network (trained on converted nuScenes data) on converted nuScenes data. However, this is not the case if I evaluate the same network (trained on converted nuScenes data) on KITTI. In that case, I get nice results (for example 3D AP for Car is 66 in Easy).

My question is, do you have any idea, what may cause such a big difference? Do you apply different operations according to dataset in your evaluation script?

I would really appreciate you answer and support. Have a nice day.

Galaxy-ZRX commented 2 years ago

Hi @kaancayli , may I ask if you have solved this problem? I am doing something quite similar and it will be helpful if you are willing to help!

To be specific, have you get reasonable results when evaluate on transferred-to-kitti-NuScenes data?

Thanks in advance for your reply and help!

kaancayli commented 2 years ago

Hi @Galaxy-ZRX , I solved the problem by using another evaluation script ("SECOND" Object detection repository). The problem is caused by a numerical error. Today, it's hard for me to dig through my browser history but I will try to share the solution with you tomorrow.

Galaxy-ZRX commented 2 years ago

Hi @kaancayli , thanks for sharing! It doesn’t matter if you can’t find it since it has been so long :), I also know that model and I will turn to it to look for some solutions. Many thanks again!

kaancayli commented 2 years ago

3D_adapt_auto_driving-evaluate.tar.gz These are files that I used for evaluation. However, I couldn't find the specific GitHub issue regarding the problem. The file eval2.py should calculate the results correctly. You might need to adjust a few path parameters according to you working environment though.

Galaxy-ZRX commented 2 years ago

@kaancayli Thank you so much!