Output RGB Value Does Not Match Value Set By simSetSegmentationObjectID()

[PatrickUtz]

I followed the instructions here for using simSetSegmentationObjectID() to set the id of my "WhiteSphere" mesh to 210. According to the docs, this should set the color of "WhiteSphere" to RGB (210, 210, 210) when taking an image using the Infrared feature with simGetImages([airsim.ImageRequest("0", airsim.ImageType.Infrared)]). However, as shown in the attached images, the sphere in the resulting .png file has a color of hex E9E9E9 or RGB (233, 233, 233). I figured that there is some offset between the desired value from simSetSegmentationObjectID() and the output .png file. Yet after setting the id of another object, I realized that the offset with its output color was different from that of the WhiteSphere and its output color.

[msb336]

@ironclownfish

Any ideas?

[ironclownfish]

@PatrickUtz This looks like the same problem that was recently fixed in https://github.com/microsoft/AirSim/pull/1881.

Can you make sure your airsim install is fully updated?

[PatrickUtz]

@ironclownfish I have verified that I am using the latest version of AirSim and have not been able to find out what the issue is. In about a week I will be porting everything over to my Windows machine and will check to see if I get the same issue. If not, I plan on checking/mapping each id to it's actual output grayscale value (so essentially make my own version of this).

[PatrickUtz]

@msb336 and @ironclownfish I figured out the problem. The reason that the pixel value I set using the simSetSegmentationObjectID() function for infrared images did not correspond to the actual grey scale value of the output image is because my settings.json file (found under /home/<username>/AirSim in linux) did not have the "CameraDefaults" setting and as such the pixel value that was set using the simSetSegmentationObjectID() function corresponded to a "random" grey scale value (not actually random...I went through the process of mapping out these "initialized" grey scale pixel values in this attached .txt file for reference segmentation_pixel_values.txt). Now that I included the "CameraDefaults" and "CaptureSettings" in the settings.json file (as shown in the screenshot below), all the pixel values I pass through the simSetSegmentationObjectID() function correspond to the same greyscale value (so simSetSegmentationObjectID(Cube1, 2) causes the Cube1 mesh in the output infrared image to have a grey scale value of [2, 2, 2]).

[espenthaem]

I'm having the same problem when using the Segmentation camera. In the Blocks environment, setting the SegmentationObjectID to 1 results in an RBG value of [199, 174, 41], instead of [153, 108, 6], according to

My settings.json looks like:

{
  "SeeDocsAt": "https://github.com/Microsoft/AirSim/blob/master/docs/settings.md",
  "SettingsVersion": 1.2,
  "SimMode": "ComputerVision",
  "CameraDefaults": {
    "CaptureSettings": [
      {
        "ImageType": 0,
        "Width": 256,
        "Height": 144
      },
      {
        "ImageType": 5,
        "Width": 256,
        "Height": 144
      }
    ]
  }
}
{
  "SeeDocsAt": "https://github.com/Microsoft/AirSim/blob/master/docs/settings.md",
  "SettingsVersion": 1.2,
  "SimMode": "ComputerVision",
  "CameraDefaults": {
    "CaptureSettings": [
      {
        "ImageType": 0,
        "Width": 256,
        "Height": 144
      },
      {
        "ImageType": 5,
        "Width": 256,
        "Height": 144
      }
    ]
  }
}

[PatrickUtz]

@epananth Try adding/playing with the "SegmentationSettings" in the settings.json. More info from the API docs here: https://github.com/microsoft/AirSim/blob/master/docs/settings.md

[espenthaem]

Thank you for your reply. Unfortunately, I'm only able to influence the ObjectID that are used to generate the mapping. However, the RGB values that I get in my SegmentationImage do not correspond to the values listed in the docs.

[PatrickUtz]

Did you add this to your settings.json and made sure to restart Unreal Editor?

[espenthaem]

Yes.

I use client.simSetSegmentationObjectID("OrangeBall", 1) in the Blocks environment, and I'm viewing the Segmentation Image using the sub window.

[PatrickUtz]

Ok, I would say try using the Infrared feature instead and see if you get the same issue.

[espenthaem]

Unfortunately I have the same issue when using the Infrared feature, I even get the same mapping of Segmentation ID's to grey scale values.

I'm using the following settings:

{
  "SeeDocsAt": "https://github.com/Microsoft/AirSim/blob/master/docs/settings.md",
  "SettingsVersion": 1.2,
  "SimMode": "ComputerVision",
  "CameraDefaults": {
    "CaptureSettings": [
      {
        "ImageType": 0,
        "Width": 256,
        "Height": 144
      },
      {
        "ImageType": 5,
        "Width": 256,
        "Height": 144
      },
      {
        "ImageType": 7,
        "Width": 256,
        "Height": 144
      }
    ]
  },
  "SegmentationSettings": {
    "InitMethod": "",
    "MeshNamingMethod": "",
    "OverrideExisting": false
  }
}

[PatrickUtz]

@espenthaem try putting "None" for the "InitMethod" section and then restart Unreal Editor. If this does not work, then I assume you are having the same issue I had and as such the infrared values are corresponding to another set of values which I mapped out here: segmentation_pixel_values.txt. The left values are the ones you set in the API and the right values are the actual infrared values that are outputted in the segmented image. Hopefully this helps!