Can I add lane/road boundaries in occupancy grid observation?

[sunningshore]

Have been trying to train continuous action space using DDPG. It is really difficult for the vehicle to stay on its lane in continuous action scenarios. Just wondering if we can add the lane/road boundaries to occupancy grid observation? Thanks!

PS: Has anyone been able to train any model in continuous action space scenarios?

[eleurent]

Hi @sunningshore,

• A first disclaimer: I haven't worked a lot with continuous actions myself beside the parking environment, which explains why some important parts may be missing in that setting, but I'd be glad to fix that.
• Some people have tried it, see https://github.com/eleurent/highway-env/issues?q=continuous, and some have succeeded
• Indeed, an important part of continuous control that is not present with discrete actions is the skill of following a road. This means that two parts must be added: i. the vehicle must be able to see the lane boundaries ii. the vehicle must be penalized when it leaves the road boundaries

So far, we discussed the part ii. in #67, by adding an offroad penalty (may not be present in every environment). The part i. was kind of handled in one particular case: a straight road (highway-v0) and the use of KinematicsObservation. Then, the vehicle's y coordinate can be used by the agent to predict whether it is driving on the road or not, and if it is going to be penalized for doing so.

Indeed, if you change the observation to the current OccupancyGrid, there is no way for the ego-vehicle to know where it should go, so this information must be added to the observation. This can be done by adding a new channel to the grid, to represent the onroad/offroad information. I will try to implement it.

Another possibility is moving from the OccupancyGrid the Grayscale Image observation, which already renders lane markings. And finally, the best solution is probably to add another kind of image-like observation, but with semantic channels, like we see in many autonomous driving papers, like ChauffeurNet. Then, we could have separate channels to render lanes, vehicles, static obstacles, etc.

[sunningshore]

Thank you for your kind explanation. _### Can I achieve part ii by setting 'offroadterminal' to True, implying offroad is penalized as collision: terminated with reward 0? If Im not wrong, I think that can also penalize offroad actions. No, I was wrong. The vehicle should be allowed to temporarily go offroad.

### Really looking forward to your part i mod!!!!!! Still, looking forward to your part i mod!!!

Thanks!

[eleurent]

Can I achieve part ii by setting 'offroad_terminal' to True, implying offroad is penalized as collision: terminated with reward 0?

Absolutely, but that config is only implemented on highway and u-turn environments, so it the corresponding code needs to be copied if you use another env. And I think that at the moment, offroad_terminal only sets the state as terminal (resets the env), but the reward is not set to 0.

No, I was wrong. The vehicle should be allowed to temporarily go offroad.

Alright :+1:

Really looking forward to your part i mod

I'll look into it today :)

[eleurent]

I implemented something which seems to be working, you can try it by installing the development version of highway-env. You need to add 'on_road' to the observation configuration's 'features' list.

[sunningshore]

Great, thanks!

[eleurent]

Here is what I get with the new on_road layer, by running SB3's PPO on @supperted825's environment proposed in #231 The reward is 1 when driving on-road, and 0 when driving off-road

https://user-images.githubusercontent.com/1706935/133697588-257530e9-da94-4d15-9e70-26e81c49f76e.mp4

[eleurent]

Just realized that the generated video is not handled by my browser. It seems to be due to a change in video recording that shipped in the latest gym release 0.20: https://github.com/openai/gym/pull/1893

I downgraded gym to 0.17 to regenerate a video:

https://user-images.githubusercontent.com/1706935/133753082-b66b4ca7-4c9a-4adf-843f-4172387d9400.mp4

[eleurent]

And here is what I get with a more sophisticated reward, that penalizes deviation from the lanes center-lines

https://user-images.githubusercontent.com/1706935/133753435-8854eec2-c78c-4c3e-813a-9b6a70722e5f.mp4

[sunningshore]

Impressive, thanks!

From: Edouard Leurent @.> Sent: Friday, September 17, 2021 20:47 To: eleurent/highway-env @.> Cc: MUXU @.>; Mention @.> Subject: Re: [eleurent/highway-env] Can I add lane/road boundaries in occupancy grid observation? (#232)

And here is what I get with a more sophisticated reward, that penalizes deviation from the lanes center-lines

https://user-images.githubusercontent.com/1706935/133753435-8854eec2-c78c-4c3e-813a-9b6a70722e5f.mp4

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/eleurent/highway-env/issues/232#issuecomment-921621799 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AD2QP656PIE6VKWZ6AEMT4LUCL6ANANCNFSM5EECIAYA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub . https://github.com/notifications/beacon/AD2QP62EV4O6JK7LYMSQPETUCL6ANA5CNFSM5EECIAYKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOG3XNKJY.gif

[nikhilkala]

@eleurent I am working on a research problem, and for that I need to reach the baseline first, which is the video that you generated. I have tried various scripts and settings, but I am unable to train and render these.

could you please share the script to how you trained this model and how do you make the model run to get this rendering video for the env -> 'racetrack-v0'?

[eleurent]

Hi, The script I used is the following: https://github.com/eleurent/highway-env/blob/abcef8da79dedf0dfdbe2db5b176d6f203366489/scripts/sb3_racetracks_ppo.py