search

opendilab / LMDrive

[CVPR 2024] LMDrive: Closed-Loop End-to-End Driving with Large Language Models
Apache License 2.0
526 stars 48 forks source link

Evaluation speed is too slow #38

Closed dingli-dean closed 3 months ago

dingli-dean commented 3 months ago

Hi, team, thanks for your attention. I try to evaluate the released model LMDrive-llava-v1.5-7b-v1.0, and observe that the evaluation process is too slow. Here is the statistics of evaluation in one route scenario, is the evaluation speed as fast as expected ? Evaluation running on one 3090 GPU.

========= Results of RouteScenario_51 (repetition 0) ------ SUCCESS =========

╒═════════════════════════════════╤═════════════════════╕ │ Start Time │ 2024-03-26 12:46:58 │ ├─────────────────────────────────┼─────────────────────┤ │ End Time │ 2024-03-26 13:58:15 │ ├─────────────────────────────────┼─────────────────────┤ │ Duration (System Time) │ 4276.92s │ ├─────────────────────────────────┼─────────────────────┤ │ Duration (Game Time) │ 324.25s │ ├─────────────────────────────────┼─────────────────────┤ │ Ratio (System Time / Game Time) │ 0.076 │ ╘═════════════════════════════════╧═════════════════════╛

╒═══════════════════════╤═════════╤═════════╕ │ Criterion │ Result │ Value │ ├───────────────────────┼─────────┼─────────┤ │ RouteCompletionTest │ SUCCESS │ 100 % │ ├───────────────────────┼─────────┼─────────┤ │ OutsideRouteLanesTest │ SUCCESS │ 0 % │ ├───────────────────────┼─────────┼─────────┤ │ CollisionTest │ SUCCESS │ 0 times │ ├───────────────────────┼─────────┼─────────┤ │ RunningRedLightTest │ SUCCESS │ 0 times │ ├───────────────────────┼─────────┼─────────┤ │ RunningStopTest │ SUCCESS │ 0 times │ ├───────────────────────┼─────────┼─────────┤ │ InRouteTest │ SUCCESS │ │ ├───────────────────────┼─────────┼─────────┤ │ AgentBlockedTest │ SUCCESS │ │ ├───────────────────────┼─────────┼─────────┤ │ Timeout │ SUCCESS │ │ ╘═══════════════════════╧═════════╧═════════╛