search

opendilab / LMDrive

[CVPR 2024] LMDrive: Closed-Loop End-to-End Driving with Large Language Models
Apache License 2.0
526 stars 48 forks source link

Evaluation on Town-05 Long benchmark #41

Open dingli-dean opened 2 months ago

dingli-dean commented 2 months ago

Hi, team. Thanks for releasing the exceptional work. I try to evaluate the released model (llava-v1.5) on town05 long benchmark (with leaderboard/data/evaluation_routes/routes_town05_long.xml), and sadly observe that the results are significantly lower than current SOTA methods. image

Did you evaluate LMDrive model on town-05 long benchmark? If so, can you show the performance comparison between LMDrive and other methods?

Thanks again for your attention, and look forward to your reply.

deepcs233 commented 2 months ago

Hi!

We haven't tested LMDrive on the other benchmarks. Our framework only tasks as input the instruction based on natural language. It's harder compared with other frameworks that can directly obtain the target point/ command. Hence, there's no sense in comparing with the traditional methods on other benchmarks.