search

dotchen / WorldOnRails

(ICCV 2021, Oral) RL and distillation in CARLA using a factorized world model
https://dotchen.github.io/world_on_rails/
MIT License
166 stars 29 forks source link

Issues with running the validation routes #24

Closed aaronh65 closed 3 years ago

aaronh65 commented 3 years ago

Hey Dian,

I've noticed an issue when deploying agents on the validation routes. On some of them, non-ego agents will be stuck in front of the ego agent which causes it to halt until timeout. The following shows route completion/driving score of the driving agent on all validation routes, each run for 5 repetitions

overall_score_metrics_0

Some of them, like route_09 and route_20 reach the same score every single time. Taking a look at route_20's debug videos, you see stuff like this

image

This vehicle just sits there for ages without doing anything, and causes the ego agent to timeout even though the ego agent didn't do anything wrong. Have you seen anything like this in your own experiments?

dotchen commented 3 years ago

Thanks for the info! The evaluation numbers are very interesting. Could you explain more on the setup?

Regarding vehicles getting stuck: This could be one of the scenarios (scenario 6 if I remember correctly), where you are expected to change lane and pass the vehicle.

aaronh65 commented 3 years ago

For evaluation, we iterate through the routes_testing folder and run each route for 4 repetitions. For each route, we record the mean and standard deviation for route completion and driving score to create the above plot.

I wasn't aware that vehicle passing was explicitly modeled in Leaderboard, I worked a lot with CARLA 0.9.11 and didn't see any of those scenarios then (not sure if it's a version issue?). I'll close this issue then if this is expected behavior

dotchen commented 3 years ago

Cool. Is this with the provided weights?

I wasn't aware that vehicle passing was explicitly modeled in Leaderboard, I worked a lot with CARLA 0.9.11 and didn't see any of those scenarios then

They are here, and here. I am not sure why you haven't seen any in 0.9.11.

aaronh65 commented 3 years ago

Ah I see, it looks like the all_towns_traffic_scenarios.json file included in Brady Zhou's LBC codebase only has scenarios 1, 3, 4... that's the codebase I built off of. I guess that explains why I never saw Scenario 06 before working with your code.

The results I posted above were not with the provided weights, I just rebenchmarked with the provided weights and this is what I see

overall_score_metrics_0

infraction_pie

dotchen commented 3 years ago

Ah I see, it looks like the all_towns_traffic_scenarios.json file included in Brady Zhou's LBC codebase only has scenarios 1, 3, 4... that's the codebase I built off of. I guess that explains why I never saw Scenario 06 before working with your code.

That makes sense. Marking as resolved.