Mismatching in screen positions calculation

[leonwu0108]

Thanks for your great work!

I was trying to implement my idea based on your code. But when I use your world2scrn() function to project the 3D points gain with the depth2wpos() function to the camera views, I noticed there was an obvious gap between the GT screen positions (pixel positions, uv) and the projected output. I want to know why this happens. I've checked your code but could not figure this out.

The functions I used: https://github.com/turandai/gaussian_surfels/blob/654511a86573f311f4875c070fef8f58791a0548/utils/image_utils.py#L222 https://github.com/turandai/gaussian_surfels/blob/654511a86573f311f4875c070fef8f58791a0548/utils/image_utils.py#L282

Hoping to get answers from you!

[turandai]

Hi, I am sorry for your problem. Could you please provide more information (an image maybe) ? So that I can help to locate the potential problem.

[leonwu0108]

I run the following steps after getting the rendered depth:

from utils.image_utils import depth2wpos, world2scrn

uv = np.mgrid[0:viewpoint_cam.image_height, 0:viewpoint_cam.image_width].astype(np.int32)
uv = torch.from_numpy(np.flip(uv, axis=0).copy()).float()
uv = uv.reshape(2, -1).transpose(1, 0)

depth_mask = ((depth > 0) & mask_gt.bool()).float()
pts_world_coords = depth2wpos(depth, mask_gt, viewpoint_cam).permute([1, 2, 0]) #(H, W, 3)
camPos, ndc, inMask, outView = world2scrn(pts_world_coords.view(-1, 3), [viewpoint_cam], 0) # ndc: (1, H*W, 2)

reso = torch.tensor([viewpoint_cam.image_width, viewpoint_cam.image_height], device=ndc.device)
scrnPos = ((ndc.squeeze(0) + 1) / 2) * reso[None, :]

uv = uv[depth_mask.squeeze(0).bool().view(-1)].long()
scrnPos = scrnPos[depth_mask.squeeze(0).bool().view(-1)].long()

and I got uv and scrnPos:

(Pdb) p uv
tensor([[ 676,    0],
        [ 677,    0],
        [ 678,    0],
        ...,
        [1568, 1199],
        [1569, 1199],
        [1570, 1199]])
(Pdb) p scrnPos
tensor([[ 962,  569],
        [ 960,  564],
        [ 945,  534],
        ...,
        [1248,  864],
        [1192,  806],
        [1045,  654]], device='cuda:0')

which are obviously mismatching.