Image Homography wrong result use camera info and DepthPlanar

[milkcat0904]

Bug report

• AirSim Version/#commit: 1.7.0
• UE/Unity version: UE4
• autopilot version: None
• OS Version: Windows Server 2019

What's the issue you encountered?

Homography is very import transformation in MVS task, the formula is

I collect two images, depth map and camera parameters. First image is the reference image, and the other is source image, I want to get homography warped source image, which is should be like reference image. But I get wrong result: | ref image | ref depth map | src image | warped src image |

I also test my image warping code in DTU dataset, the result is right: so, I speculate that there is something wrong with the camera parameters and depth map of the airsim

Settings

"Vehicles": {
    "SimpleFlight": {
      "VehicleType": "SimpleFlight",
      "Cameras": {
        "f_center": {
          "X": 0.46,
          "Y": 0,
          "Z": 0.675,
          "Pitch": 0.0,
          "Roll": 0.0,
          "Yaw": 0.0,
          "CaptureSettings": [
            {
              "ImageType": 0,
              "Width": 500,
              "Height": 500,
              "FOV_Degrees": 82
            },
            {
              "ImageType": 1,
              "Width": 500,
              "Height": 500,
              "FOV_Degrees": 82
            },
            {
              "ImageType": 2,
              "Width": 500,
              "Height": 500,
              "FOV_Degrees": 82
            },
            {
              "ImageType": 5,
              "Width": 500,
              "Height": 500,
              "FOV_Degrees": 82
            }
          ]
        }
      },

How can the issue be reproduced?

I use airsim.ImageType.DepthPlanar as depth map

 responses = client.simGetImages([
        airsim.ImageRequest(camera_name, airsim.ImageType.Scene, False, False),
        airsim.ImageRequest(camera_name, airsim.ImageType.DepthPlanar, True, False)
    ])  # scene vision image in png format

 airsim.write_pfm(depth_file, airsim.get_pfm_array(responses[1])) # save pfm file

i use client.simGetCameraInfo to get camera parameters

camera_name = 'f_center'
client.simSetCameraFov(camera_name, 82) # set camera fov

client.moveToPositionAsync(0, 0, -3, 1).join() # reference image drone's location 
ref_camera_info = client.simGetCameraInfo(camera_name)

client.moveToPositionAsync(0, 2, -3, 1).join() # source image drone's location
src_camera_info = client.simGetCameraInfo(camera_name)

calculate camera intrinsics and extrinsics

cam_position = camera_info.pose.position # xyz
cam_orient = camera_info.pose.orientation # wxyz
cam_fov = 82
cam_resolusion = (500, 500)

# intrinsics
w, h = cam_resolusion[0], cam_resolusion[1]
cx, cy = w/2, h/2
fx = w /(2*math.tan(math.radians(cam_fov/2)))
fy = h /(2*math.tan(math.radians(cam_fov/2)))

intr_mat = np.array([
    [fx, 0, cx], [0, fy, cy], [0, 0, 1]
])

# extrinsics
q0, q1, q2, q3 = cam_orient[0], cam_orient[1], cam_orient[2], cam_orient[3]

extr_mat = np.eye(4)

r = R.from_quat([q1, q2, q3, q0])
extr_mat[:3, :3] = r.as_matrix()

# translation
extr_mat[0, 3] = cam_position[0]
extr_mat[1, 3] = cam_position[1]
extr_mat[2, 3] = cam_position[2]

Include full error message in text form

What's better than filing an issue? Filing a pull request :).

[lnexenl]

1. I am a little confused. You say "homography warped source image, which is should be like reference image.", but in your formula, q_{xyz} in is 3d coordinate of points in source image. So homography warped image should be like source image? (I don't know whether I have a correct understanding)
2. In your intrinsics' calculation, you miswrite 82 for 92, this should have some effects on your result.

[milkcat0904]

1. I am a little confused. You say "homography warped source image, which is should be like reference image.", but in your formula, q_{xyz} in is 3d coordinate of points in source image. So homography warped image should be like source image? (I don't know whether I have a correct understanding)
2. In your intrinsics' calculation, you miswrite 82 for 92, this should have some effects on your result.

• I use cv2.remap for warping

  warped = cv2.remap(src_image, y, x, interpolation=cv2.INTER_LINEAR)

• the fov is 82 in my code, it's just miswriting on websites, with no effects on my result.

[Leon-xc]

I found something wrong as well. The output of client.simGetCameraInfo has proj_mat which is like [[0.0, 1.0, 0.0, 0.0], [0.0, 0.0, -1.7777777910232544, 0.0], [0.0, 0.0, 0.0, 10.0], [-1.0, 0.0, 0.0, 0.0]]. but this project_mat is not like a standard projection matrix. Is there anything wrong with this API function (client.simGetCameraInfo) ?