image distortion whenever the camera view is not perpendicular to the walls

[chch9907]

I am using the depth image and camera intrinsics and extrinsics to reconstruct the 3D point cloud in scene "FloorPlan212" . Initializing the camera with yaw of 0° in the world coordinate, when I rotate 0°, 90°,180°, 270° or nx90°, e.g., rotate 0° in following image, the reconstruced point cloud can show that the corners of the house are correctly perpendicular.

However, when I rotate any other angles, e.g. 60°, both the following rgb image and the reconstructed point cloud indicate the corners are not perpendicular, which may result from some distortion and will cause wrong top-down projection.

For reference, the official floor map of scene "FloorPlan212" is attached as follows.

The potential image distortion issue has confused us for a week until comparing the results with that in habitat. I think this issue will constrain the rotation space as well as the yaw angle at the initial position.

I wonder if this issue indeed exists and if so is there any solution? Thanks

[Lucaweihs]

Hi @chch9907, looking at the provided point clouds, it seems likely that the camera's FOV is not being correctly taken into account. One important thing to keep in mind is that the FOV reported by AI2-THOR is the vertical FOV and not the horizontal FOV. We have some code in the allenact library (see here) that generates point clouds from depth images (note that the documentation for the linked function says it only works for square images but I believe it should actually work for images of any size).

[chch9907]

Hi @Lucaweihs , thank you for your reply! Indeed it came from the wrong FOV. In fact we have tried to consider it as vertical FOV. Today we check the code again and find a bug. Now we can generate correct results. Thank you so much for your reminder again!