microsoft / AirSim

Open source simulator for autonomous vehicles built on Unreal Engine / Unity, from Microsoft AI & Research
https://microsoft.github.io/AirSim/
Other
16.43k stars 4.58k forks source link

Holes in depthmap #4610

Open DonMaEn opened 2 years ago

DonMaEn commented 2 years ago

Bug report

What's the issue you encountered?

When using the orthographic camera, holes appear in the depth map images saved to disk when using the client.startRecording() function. These holes only appear on the ground mesh, not on buildings or trees. I have tested this using the pre-compiled AirSimNH scene and the LandscapeMountains scene. In the below images you can see in yellow the hole, and you can see how the trees are not affected by this bug. The RGB image is from the exact same area as the depthmap image, and you can see there is no visible hole in the ground mesh.

image

Also, see the following screen shot of a point cloud I created from the depth maps and RGB images that shows where the holes appear in the AirSimNH scene: image

Settings

{
  "SeeDocsAt": "https://github.com/Microsoft/AirSim/blob/master/docs/settings.md",
  "SettingsVersion": 1.2,
  "SimMode": "ComputerVision",
  "Recording": {
    "RecordOnMove": true,
    "RecordInterval": 0.05,
    "Cameras": [
        { "CameraName": "0", "ImageType": 0, "PixelsAsFloat": false, "Compress": true },
        { "CameraName": "0", "ImageType": 5, "PixelsAsFloat": false, "Compress": true },
        { "CameraName": "0", "ImageType": 1, "PixelsAsFloat": true, "Compress": false }
    ]
  },
  "CameraDefaults": {
    "CaptureSettings": [
      {
        "ImageType": 0,
        "Width": 728,
        "Height": 728,
        "OrthoWidth": 60,
        "AutoExposureSpeed": 100,
        "MotionBlurAmount": 0,
        "ProjectionMode": "orthographic"
      },
      {
        "ImageType": 1,
        "Width": 728,
        "Height": 728,
        "OrthoWidth": 60,
        "AutoExposureSpeed": 100,
        "MotionBlurAmount": 0,
        "ProjectionMode": "orthographic"
      },
      {
        "ImageType": 5,
        "Width": 728,
        "Height": 728,
        "OrthoWidth": 60,
        "AutoExposureSpeed": 100,
        "MotionBlurAmount": 0,
        "ProjectionMode": "orthographic"
      }
    ]
  },
  "SubWindows": [
    {"WindowID": 0, "CameraName": "0", "ImageType": 3, "VehicleName": "", "Visible": true},
    {"WindowID": 1, "CameraName": "0", "ImageType": 5, "VehicleName": "", "Visible": true},
    {"WindowID": 2, "CameraName": "0", "ImageType": 0, "VehicleName": "", "Visible": true}
  ],
  "SegmentationSettings": {
    "InitMethod": "",
    "MeshNamingMethod": "",
    "OverrideExisting": false
  }
}

How can the issue be reproduced?

  1. Use the pre-compiled AirSimNH scene and the above settings file
  2. Collect images using the airsim record feature in a lawnmower style pattern, similar to the following script: https://github.com/MizzouINDFUL/UEUAVSim/blob/main/airsim/airsim_collect.py
  3. Collect the images with pitch = -90, i.e. directly downward

Include full error message in text form

N/A

Notes:

Because I am using centos stream 8 I have not been able to compile Airsim from source, and therefore could not test on the master branch.

I tried using the latest version 1.8 pre-compiled binaries, however when running my collection script airsim crashes. I'm guessing this is because I am still using version 1.7 of the airsim python API. 1.8 is not yet available on pypi so I cannot test this theory.

jonyMarino commented 2 years ago

Thanks for the report @DonMaEn! The holes always appear in the same places or this is a stochastic behavior?

DonMaEn commented 2 years ago

@jonyMarino, good question. The holes do not always appear in the same place. The following image is a point cloud I created using the same method, but of a smaller area. You can still see the holes, but they are in different locations. image

artths commented 1 year ago

In my case, holes also appear randomly on RGB camera images (Scene) in the form of randomly shaped white spots, on DepthPerspective it is a spot with a constant maximum value of 65k. DepthVis frame is completely black with the exception of random white spots. This mainly happens when the drone turns. I don't notice it in the simulator itself. I use the latest official build for Windows. I tried to run the simulator on both Nvidia and AMD GPUs.

artths commented 1 year ago

OK found a way around the problem. Repeating the same query produces normal images.