Which scenario file was used for evaluation?

[Kait0]

Hi,

I wanted to know which scenario.json file was used for the ablations in this paper. I couldn't find the information in the README. The leaderboard version linked in the repository is from Jan 2021 it has one scenario file (all_towns_traffic_scenarios_public.json), but there seem to be only scenarios 1,3 and 4 specified within them. In the current master branch of the leaderboard there is also an all_towns_traffic_scenarios_public.json that additionally contains scenarios 2,7,8,9,10.

So, I am curious which scenario file was used for training and evaluation in this work (and which types of scenarios were considered)

[dotchen]

Hi @Kait0,

The scenario file used for eval and local ablation comes from the linked leaderboard repo, which is the stable branch, the one instructed to use for leaderboard submission. Training scenario follows Rails where we use the larger scenarios file. We filter and only use scenarios 1,3,4,7,8,9,10, same as Rails. It essentially uses the same data collection scheme as Rails but uses a slightly modified version of the Carla official navigation agent in 0.9.10.1 (fixing for the pedestrian and some off traffic light stopping).

[Kait0]

thanks for the answer.

[dotchen]

Thanks for the reminder also, I previously had not realized the one provided by leaderboard/stable and scenario_runner are so different... I double-checked and the above is correct -- eval/ablation uses the one specified by the environment variable which is the one linked in the leaderboard here, and data collection uses a specified path pointing to the scenario file, which is copied from the scenario_runner/leaderboard. (seems also different than the one currently in the master branch)