Questions about longest6 evaluation

[yihang99]

Hello, I had one more question about this if you don't mind.

The 1st question: In my comprehension of your answer of previous issues, longest6 sets vehicle number to be maximum of each town, and contains no pedestrians. But in the graph of your paper there exists the Ped term. Does this mean you set a fraction of the spawn points to be pedestrians? (If that is true, could you please tell me where this amount is set in the codebase?)

The 2nd question: Comparing Table6 and 7 in the paper, I see that the column 'Stop' Exists only in Table7. Does that mean longest 6 does not set penalty for stop sign infractions?

The 3rd question: I wonder if you can release some more detailed statistics of some some of your key experiments? (Such as the .csv file produced by _resultparser.py, or the evaluation videos if saved them) I am using other simple methods to test on longest6 and I noticed that some of the routes are extremely hard, such as _RouteScenario13, where the agent needs to turn around at a roundabout to reach a target point in the backward side of the ego vehicle. I am curious about the performance on specific routes of your sota method. (The 69 point version in the graph.)

[yihang99]

Sorry, for the second question, I just saw the relative parts and got clear about that. I'll be very grateful if you can answer the other 2 question. : )

[Kait0]

There are no pedestrian spawn points/ regularly walking around, but pedestrians appear as part of scenario 3 (which involves a pedestrian hidden behind an object running in front of your car). So the methods need to be able to stop for pedestrians and there is a pedestrian collision infraction.

Will answer the other question later.

[yihang99]

Thanks for your reply! However I am still not quite clear yet. As far as I know, longest6 contains 6 routes of 6 towns each, with indices from 0 to 35. Which of them (during evaluation) corresponds to that scenario3 that you mentioned? (Up to now I only found the name 'scenario3' in the training route directory, as a subset of training route set.)

[Kait0]

Scenarios are specified in the following file. You need to specify it when running the eval (see readme)

All of the routes could potentially included the scenario (multiple times). Don't know by heart where it occurs.

[yihang99]

IC, thanks for you reply! My last question is about detailed evaluation statistics or video as I said in my first message. I hope you can release some of them in a proper type if possible. Anyway, thanks for your patient answer to all my questions!

[Kait0]

The result files for the 69 DS version in Table 6 that you mentioned above: table6_69_model.zip

[yihang99]

Thank you very much!