Applied to autonomous driving

[StephenGordan]

Can DreamerV3 be used in self-driving cars? Are there any related works for reference?

[Hamptonjc]

Somewhat related: https://arxiv.org/abs/2309.17080

AFAIK Comma.ai is also exploring the use of world models to train a driving policy.

[gaodechen]

Yes!

Feel free to check out our recent project and demos at https://github.com/ucd-dare/CarDreamer and there is a technical report for more details

Dreamer is a fantastic toolbox to explore RL for self-driving. We recently made some training and evaluation tasks, such as right turn, left turn, roundabout, overtake, for self-driving in CARLA. It may take only 1-4 hours for DreamerV3 to master each task which was quite amazing considering it learns the policy from scratch in imagination.

- Check out demos from bird's eye views

Roundabout	Right Turn	Overtake

- CARLA supports highly realistic rendering and sensors like cameras, LiDARs

Left Turn Hard Camera	Right Turn Hard Camera	Roundabout Camera

• Another interesting fact is that by sharing agents' intentions (waypoints as rendered in the bird's eye views), the ego vehicle can perform significantly better in tasks like left turn and right turn. We spawned random traffic flows that are going to different directions and therefore the ego vehicle has to determine whether a vehicle of interest is interfering with it in the future. DreamerV3 can successfully learn the implications of shared waypoints, and cut in the traffic flow safely and decisively at crossroads!

Right Turn Hard with Shared Intentions	Right Turn Hard w/o Sharing Intentions