search

Thinklab-SJTU / Bench2Drive

[NeurIPS 2024 Datasets and Benchmarks Track] Closed-Loop E2E-AD Benchmark Enhanced by World Model RL Expert
Other
1.29k stars 85 forks source link

Question about resume #85

Closed YiHuang108 closed 2 months ago

YiHuang108 commented 2 months ago

Sometimes, when I run the run_evaluation_Multi script again after the system breaks down, the script will continue to run from the middle. For example, when I run the 55th/74th script, the script will be resumed from the 30th script. This problem persists even if I directly change the number of processes.

jayyoung0802 commented 2 months ago

For UniAD, There are a few things to pay attention to，

  1. TASK_NUM == len(TASK_LIST)==len(GPU_RANK_LIST)== how many processes
  2. split_xml.py run only allowed once. Otherwise, if you made the changes according to 3, it will be replaced.
  3. If one route crashed, you need manually subtract 1 from result.json and comment the route in xml.
YiHuang108 commented 2 months ago

I didn't do anything with the .xml file, and I only changed the count to the correct number in the .json file after the error occurred accidentally. Sometimes I can resume the process by rerunning the script; other times, it resume in the middle of the procedure.

jayyoung0802 commented 2 months ago

When you change the count to the correct number in the .json file, you need to comment this crash route in xml. len(route in xml) == count (in the .json file)

jiaxiaosong1002 commented 2 months ago

@YiHuang108 You may read the code in https://github.com/Thinklab-SJTU/Bench2Drive/blob/main/leaderboard/leaderboard/leaderboard_evaluator.py to understand the logic of resume code.

starlighttt123 commented 2 months ago

For UniAD, There are a few things to pay attention to，

  1. TASK_NUM == len(TASK_LIST)==len(GPU_RANK_LIST)== how many processes
  2. split_xml.py run only allowed once. Otherwise, if you made the changes according to 3, it will be replaced.
  3. If one route crashed, you need manually subtract 1 from result.json and comment the route in xml.

hello, I'm a little confused of the No.3 Point, could you please give a more detail description of it? 3. If one route crashed, you need manually subtract 1 from result.json and comment the route in xml. btw, if for vad, it's still work?

jayyoung0802 commented 2 months ago

Similar to #89