Surface normal from global coordinate to camera coordinate

[fuenwang]

Hi, thanks for releasing this amazing dataset. From the README, the surface normal of equirectangular image is in global coordinate. But I want to calibrate them into camera coordinate. I have checked the pose file but the rotation in it seems not consistent with the coordinate. Could you tell me how to do this conversion?

Thanks!

[meder411]

Perhaps I'm oversimplifying, but I believe you just need to rotate the normals by the transpose of the rotation matrix in the pose file (the left 3x3 R matrix from the RT matrix provided). I tried both R and R-transpose, and the latter seemed to make more sense. For example, the ceiling pixels went from ~[0,1,0] to ~[0,0,-1] and the floor pixels were the negative of those. Just applying R (no transpose) gives a far less axis-aligned output, which would be odd for the ceiling and floor normals.

Sample:

# n := normals as (H,W,3) loaded (and converted) from PNG
# rt := RT matrix as (3,4) loaded from JSON pose file

import numpy as np
new_n = np.matmul(rt[:,:3].transpose(1,0), n[...,None]).squeeze()

# new_n is (H,W,3) rotated normals

Perhaps @alexsax can confirm?

[fuenwang]

Now, my way is to just ignore z-axis, And the x-axis rotation in the file will be applied to minus y-axis rotation of the normal image, For now, they looks correct but I cannot sure about it.

[alexsax]

@meder411 Yes I think that's correct. The RT matrix in the JSON file came from Blender's matrix_world. I believe that is the matrix from camera coordinates -> global coordinates, and so using R transpose is exactly what you want @fuenwang.

Thanks @meder411!