Issues related to doing reinforcement learning in the Airsim environments - location markers

[kaihuchen]

As stated in another comment thread, I am looking to do some reinforcement learning (RL) experiments in the Airsim environments (starting with using a car in the Neighborhood environment). In order for RL to work I need to define proper penalties and rewards for the car. For the last experiment that I did (see video here https://www.youtube.com/watch?v=InrQgdU8rQs&feature=youtu.be ）I took a shortcut by using speed and collision as the reward/penalty, which is easy for me to write the code to compute and enforce. Next I want to enforce more complex rules, such as penalizing the car when it drives over the lawn/walkway/etc. In order to achieve this, my reward-enforcing code needs a way to recognize when the car is over the lawn (or the walkway, driveway, etc.)

My question here is: what's the best way for my code to detect those forbidden areas? Are there some kind of location markers in an Airsim/Unreal environment that I can access in order to extract some kind of property description for a given location (like: this location X is part of a lawn)? Having such information will help me automate the entire training process, so that I don't have to manually supervise the learning process every step of the way.

[sytelus]

There are two ways of do it, I think.

1. You can measure the width of the road and then based on position you can penalize. One way to accomplish this is to use ";" key that will show you debug report that shows you coordinates of the vehicle. So for specified path you may have set of "permissible" coordinates that vehicle should be confined to.

2. Second way is to use segmentation view to recognize the road. Currently, the segmentation works is by assigning integer value to each mesh in Unreal environment. If you search for "setStencilIDs" then you will find the function that basically assigns each mesh some random integers. I'm adding an API to be able to set this from Python but if you can't wait then you can just modify this function to assign your some integer value to all road meshes and for rest you can assign some other. This way you can get segmentation view that only shows road with some color. This can let you identify easily if you are off the road.

[kaihuchen]

@sytelus Much appreciated for your info!

FYI, I tried the ';' keyboard command, and found that it works in the Drone mode, but in the Car mode it gives me only a blank rectangle.

[ashkapoor]

Or do imitation learning

Sent from my iPhone

On Oct 12, 2017, at 8:22 AM, Shital Shah notifications@github.com wrote:

There are two ways of do it, I think.

You can measure the width of the road and then based on position you can penalize. One way to accomplish this is to use ";" key that will show you debug report that shows you coordinates of the vehicle. So for specified path you may have set of "permissible" coordinates that vehicle should be confined to.

Second way is to use segmentation view to recognize the road. Currently, the segmentation works is by assigning integer value to each mesh in Unreal environment. If you search for "setStencilIDs" then you will find the function that basically assigns each mesh some random integers. I'm adding an API to be able to set this from Python but if you can't wait then you can just modify this function to assign your some integer value to all road meshes and for rest you can assign some other. This way you can get segmentation view that only shows road with some color. This can let you identify easily if you are off the road.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[sytelus]

I have just pushed the fix for debug info showing blank (";" key).

[AloshkaD]

@sytelus do you think you can elaborate on point 2 "Second way is to use segmentation view to recognize the road. Currently, the segmentation......". For instance, if a neural network can segment the road how the road mesh can be assigned an integer and how could we use that integer to keep controller on the road? Have you managed to introduce the API for Python that you referenced above? Thanks!

[NextSim]

@AloshkaD Did you ever get this working?

We are working on reinforcement learning in the neighborhood environment, but after days of training the car does not manage to maneuver around the environment very well and will either miss turns or run into cars.

Have you been able to train it so it can just drive for several minutes and stay on the road?

[AloshkaD]

@NextSim sorry about the late response, I didn't get a notification for this. The answer is no I didn't, I'm using drone for reinforcement learning and there I have other constraints.