Semantic Segmentation

[bhairavmehta95]

To do some cool computer vision tasks, some students in the UdeM course have discussed semantic segmentation. For example:

(Imagine the whole image was segmented, not just the house)

Here, each color corresponds to a different object in the scene.

In my opinion, the easiest way to do this is to change the PNG images themselves - Description of method:

Go into gym_duckietown/textures, and then replace the objects in particular that you're looking for with simple color block pngs.

So, you'd replaces house.png with a png of a red square, and then the texture will then pull it.

To do this a bit more cleanly, i'd create a new directory called gym_duckietown/color_textures and then have a if statement here:

https://github.com/duckietown/gym-duckietown/blob/0f8fc8c19b395061cad12673280845e7e80bb19e/gym_duckietown/objmesh.py#L225

That tells you which directory to pick from. I.e if you enable color segmentation mode, then you pick from the directory called color_textures.

We'd also want to turn off lighting, which you can do by commenting out these lines:

https://github.com/duckietown/gym-duckietown/blob/0f8fc8c19b395061cad12673280845e7e80bb19e/gym_duckietown/simulator.py#L343-L348

Normally, I'd advocate for a programmatic method, but really, we don't have that many unique objects - so it's a one-time thing. And, this method would allow us to do semantic segmentation on the road tiles themselves (color the png), which we wouldn't be able to with any sort of Python method.

I can work on the master solution as well, but I'd be more than happy to merge any solution made by the students: @placaille @lavoiems @krrish94

@maximecb Do you have any thoughts / alternative approaches, especially if we want to do semantic segmentation on things like lanes on the road as well?

[maximecb]

I think that to disable lighting, you can just comment out the line that says glEnable(GL_LIGHTING), or add an extra line that says glDisable(GL_LIGHTING) after.

Changing the textures seems like an easy way to go. Programmatically, it's also possible to do glDisable(GL_TEXTURE_2D) in strategic places and then use glColor, but that seems like more work.

To segment the lanes, you could edit the road textures and adjust them. You'd want to make sure domain randomization is turned off, in which case it will always pick the first texture available.

[liampaull]

code for the resulting project is now forked into Duckietown org.

[maximecb]

@liampaull Amazing that your students got that working, the videos look great! :)