search

Thinklab-SJTU / Bench2Drive

Cloased-Loop E2E-AD Benchmark Enhanced by World Model RL Expert
Apache License 2.0
1.13k stars 78 forks source link

Question about resume #85

Open YiHuang108 opened 1 week ago

YiHuang108 commented 1 week ago

Sometimes, when I run the run_evaluation_Multi script again after the system breaks down, the script will continue to run from the middle. For example, when I run the 55th/74th script, the script will be resumed from the 30th script. This problem persists even if I directly change the number of processes.

jayyoung0802 commented 1 week ago

For UniAD, There are a few things to pay attention to，

  1. TASK_NUM == len(TASK_LIST)==len(GPU_RANK_LIST)== how many processes
  2. split_xml.py run only allowed once. Otherwise, if you made the changes according to 3, it will be replaced.
  3. If one route crashed, you need manually subtract 1 from result.json and comment the route in xml.
YiHuang108 commented 1 week ago

I didn't do anything with the .xml file, and I only changed the count to the correct number in the .json file after the error occurred accidentally. Sometimes I can resume the process by rerunning the script; other times, it resume in the middle of the procedure.

jayyoung0802 commented 1 week ago

When you change the count to the correct number in the .json file, you need to comment this crash route in xml. len(route in xml) == count (in the .json file)

jiaxiaosong1002 commented 1 week ago

@YiHuang108 You may read the code in https://github.com/Thinklab-SJTU/Bench2Drive/blob/main/leaderboard/leaderboard/leaderboard_evaluator.py to understand the logic of resume code.

starlighttt123 commented 1 week ago

For UniAD, There are a few things to pay attention to，

  1. TASK_NUM == len(TASK_LIST)==len(GPU_RANK_LIST)== how many processes
  2. split_xml.py run only allowed once. Otherwise, if you made the changes according to 3, it will be replaced.
  3. If one route crashed, you need manually subtract 1 from result.json and comment the route in xml.

hello, I'm a little confused of the No.3 Point, could you please give a more detail description of it? 3. If one route crashed, you need manually subtract 1 from result.json and comment the route in xml. btw, if for vad, it's still work?

jayyoung0802 commented 1 week ago

Similar to #89