search

Thinklab-SJTU / Bench2Drive

[NeurIPS 2024 Datasets and Benchmarks Track] Closed-Loop E2E-AD Benchmark Enhanced by World Model RL Expert
Other
1.3k stars 85 forks source link

The CARLA simulator experiences a crash when the autonomous vehicle remains stationary for an extended period #86

Closed AvinsWang closed 2 months ago

AvinsWang commented 2 months ago

Your work has set a benchmark in closed-loop evaluation for end-to-end autonomous driving methods, offering researchers a unified and fair evaluation framework that facilitates easier model assessment and performance comparison.

Despite the impressive contributions, there seem to be some issues. While using the code to closed-loop evaluate my own model, I encountered a situation where erroneous planning instructions from the model caused the ego vehicle to remain stationary for an extended period. After a certain duration, the Carla simulator crashed and threw a Timeout exception. I believe this behavior is reasonable, otherwise, the simulator would continue running. Curiously, this exception wasn't caught within the thread, leading to a crash in the closed-loop evaluation and an empty final result. I would like to understand the following:

  1. Under what conditions does the Carla simulator crash?
  2. How does the Carla simulator detect that the ego vehicle has remained stationary for an extended period?
  3. How can the above issue be resolved?
jiaxiaosong1002 commented 2 months ago

@AvinsWang You should read the code carefully in https://github.com/Thinklab-SJTU/Bench2Drive/blob/main/leaderboard/leaderboard/leaderboard_evaluator.py to understand the logic of closed-loop evaluation.