questions about results

[jessapinkman]

hey! I noticed that in open-loop simulations, STR achieved better performance because evaluation metrics like FDE and ADE are commonly used for prediction tasks. However, in closed-loop simulations, STR performed poorly. I do not deny that STR shows good performance on prediction tasks, but on the nuplan dataset, a good prediction model cannot be converted into closed-loop capabilities. I'm confused about the huge gap between CLS and OLS. Can you provide some more insights into the performance of STR on the nuplan dataset? THANKS!

[larksq]

Hi there,

I would like to share this paper "Parting with Misconceptions about Learning-based Vehicle Motion Planning" by the team of Prof. Andreas Geiger with you for your question. They have discussed this gap in depth and provided a simple yet effective way to solve it by providing IDM planning results as references during training.

Don't hesitate to contact me by email if you have more questions or ideas in mind to discuss.

[jessapinkman]

Thanks for the replay. Well, that is the shortness of the nuplan dataset. Another question: Have you ever tried to use simple post-solve technology for STR? Is this obvious help for STR? Although I know you may want to build a pure learning model.

[larksq]

We did make some very limited tries and post-solvers did help especially when the models were weak. You can implement and try some during the generation process like this off-road checking codes (which is under work-in-progress condition). PRs are welcome!

[jessapinkman]

Alright, thanks a lot for your kind reply and contribution. I will close it.