search

Thinklab-SJTU / Bench2Drive

[NeurIPS 2024 Datasets and Benchmarks Track] Closed-Loop E2E-AD Benchmark Enhanced by World Model RL Expert
Other
1.3k stars 85 forks source link

Question about the results #49

Closed YiHuang108 closed 3 months ago

YiHuang108 commented 3 months ago

Thank you very much for your outstanding work！

This is returned when I run the evaluation code, and I want to know the meaning of each line:

========= Results of RouteScenario_1773 (repetition 0) ------ FAILURE =========

╒═══════════════════════╤═════════════════════╕
│ Start Time            │ 2024-08-08 11:21:52 │
├───────────────────────┼─────────────────────┤
│ End Time              │ 2024-08-08 11:40:56 │
├───────────────────────┼─────────────────────┤
│ System Time           │ 1143.51s            │
├───────────────────────┼─────────────────────┤
│ Game Time             │ 100.05s             │
├───────────────────────┼─────────────────────┤
│ Ratio (Game / System) │ 0.087               │
╘═══════════════════════╧═════════════════════╛

╒═══════════════════════╤═════════╤═════════╕
│ Criterion             │ Result  │ Value   │
├───────────────────────┼─────────┼─────────┤
│ RouteCompletionTest   │ FAILURE │ 27.31 % │
├───────────────────────┼─────────┼─────────┤
│ OutsideRouteLanesTest │ SUCCESS │ 0 %     │
├───────────────────────┼─────────┼─────────┤
│ CollisionTest         │ SUCCESS │ 0 times │
├───────────────────────┼─────────┼─────────┤
│ RunningRedLightTest   │ SUCCESS │ 0 times │
├───────────────────────┼─────────┼─────────┤
│ RunningStopTest       │ SUCCESS │ 0 times │
├───────────────────────┼─────────┼─────────┤
│ MinSpeedTest          │ FAILURE │ 87.27 % │
├───────────────────────┼─────────┼─────────┤
│ InRouteTest           │ SUCCESS │         │
├───────────────────────┼─────────┼─────────┤
│ AgentBlockedTest      │ SUCCESS │         │
├───────────────────────┼─────────┼─────────┤
│ ScenarioTimeoutTest   │ SUCCESS │ 0 times │
├───────────────────────┼─────────┼─────────┤
│ Timeout               │ SUCCESS │         │
╘═══════════════════════╧═════════╧═════════╛

> Registering the route statistics
False

Thx very much!

jayyoung0802 commented 3 months ago

It means 27.31% of the route completed. The agent did not drive off the road, did not run a red light/stop sign. But the task is over, maybe Carla reported an error due to some system reasons.

YiHuang108 commented 3 months ago

Thx very much!