Get annotation images

[Maxfashko]

Hello! How can I get annotated images? What steps should I take to set up the annotation camera?

[lovettchris]

Do you want ground truth, or are you looking to test an image recognition engine? There is currently no built in support for automatic annotation. One idea is to take the images produced by AirSim and send them to your favorite image recognition service, like the Microsoft computer vision api, which finds what is in the image like this:

or if you like the google vision api:

[Maxfashko]

lovettchris, Microsoft computer vision api оnly provides a text label for the class. I need segmentation images. In this video (minute 6), the segmentation camera displays an annotated image painted with different colors https://www.youtube.com/watch?v=HNWdYrtw3f0&feature=youtu.be

[sytelus]

Here's some background on how ground truth segmentation in AirSim works. Each environment in Unreal contains lots of meshes representing objects in that environment like car, houses, trees etc. We assign each mesh in Unreal environment is assigned value from 0 to 255 and that value is mapped to some color. If you want to focus on certain objects like trees, you can assign specific integer value for those meshes. Right now, by default we just assign random values to each mesh. To change this, currently you have to change code. In future, we will enable API for this task. Below are some details on how to do this.

These 0-255 number assigned to each mesh is called stencil IDs. The code for setting stencil IDs is in FlyingPawn.cpp in function AFlyingPawn::setStencilIDs. If you look at this 10 lines of code, what it does is to find all the components in the environment and then assign a random 0-255 stencil ID. Now lets say you if you are interested only in trees and you want to paint them as red. Then one way to achieve this is just remove the code in AFlyingPawn::setStencilIDs (i.e. make it empty function. Then go to Unreal editor and search for tree in World Outliner. You will 28 different types of trees available there. You can select the one that interests you and then go to Details tab, Rendering section, enable "Render CustomDepth Pass" and set "CustomDepth Stencil Value" to say 1. Below is the screenshot:

[Maxfashko]

Hi, @sytelus. Was it developed api for this method?

[abhay-venkatesh]

FlyingPawn.cpp is now gone from the unreal/plugins folder?

Furthermore, on building the plugins from scratch, I cannot get any segmentation at all, I get a blank image.

[sytelus]

There has been lots of refactoring. FlyingPwn got moved in vehicles/multirotor folder. If you get latest, rebuild things should work fine. If you see issue, delete your settings.json as it might have some bad settings.

[wujiyoung]

@sytelus Is it able to access the segmentation label in the latest releases CityEnviron (for example) ? I can archive a segmentation image by ImageRequest(0, ImageType::Segmentation), but I can not define the label class of it.

[PatrickUtz]

@sytelus, does this imply that you can only label a max of 255 meshes/objects per frame?