search

dotchen / WorldOnRails

(ICCV 2021, Oral) RL and distillation in CARLA using a factorized world model
https://dotchen.github.io/world_on_rails/
MIT License
167 stars 29 forks source link

Ground truth appears to be occasionally incorrect at intersections #23

Closed aaronh65 closed 3 years ago

aaronh65 commented 3 years ago

Hi Dian,

It sometimes looks like the ground truth computed from by the action labeler gives incorrect supervision at red lights. See the below example:

media_images_train_image_85_a9a91a81

I don't think I've changed any of the action labeling logic, but it looks like the ground truth assigns a very low probability to braking even though that would be the correct action in this situation. Have you seen anything like this/have any suggestions for fixing the issue? In general, it looks like red light infractions make up the vast majority of infractions, and they often lead to other infractions like vehicle collisions or route deviations.

infraction_pie

dotchen commented 3 years ago

Do you see this all the time for traffic lights?

If so you definitely changed the traffic light related logic.

aaronh65 commented 3 years ago

diffing bellman.py between your code and mine shows no differences as far as I can tell. How large are the no speed reward zones for traffic lights when they are yellow/red? Do they extend into the intersections?

At test time, the agent will obey some of the traffic lights and wait until they turn green. I don't have an exact number of # TLs successfully passed vs # TLs failed but the agent does not run every traffic light.

dotchen commented 3 years ago

Thanks for the clarification. There are several things going on here.

First, when you mention traffic light infraction makes up most of the error, do you mean the image agent you trained, or the image agent with the downloaded weights (provided in this repo), or the Q agent?

Second, do you see examples posted above everywhere, in every town? Or does it mostly appear in Town10HD?

One thing I am sure: traffic light infraction by default does not make up most of the infractions, and this is also shown in our public leaderboard entry's metrics.

aaronh65 commented 3 years ago

I've benchmarked both the pretrained weights and my own trained weights (initialized from pretrained). They both appear to have this red light infraction distribution. Just to be absolutely certain, I'm rerunning the pretrained weights on routes_testing right now in case I messed something up.

I don't want to put extra work on your shoulders, but do you happen to have results from the routes_testing split available? Sometimes I suspect that the training/testing/held-out routes have weird distributions through towns and stuff.

dotchen commented 3 years ago

Sometimes I suspect that the training/testing/held-out routes have weird distributions through towns and stuff.

Could be true, but I currently do not have spare compute to verify this.

Generally, I do not think there is a problem for traffic lights infraction (see table 6 of paper).

However, in the later US towns (Town10HD, for example), the traffic light trigger box definition is different for certain intersections, and if the vehicle is deep into the intersection, it is out of the trigger box even if you see a red light in the RGB image. And I don't think you want the to stop in that situation either: the crossing traffic will hit the ego vehicle if it blocks in the intersection.