How to sample cube map texture?

[yifanlu0227]

Hi! Thanks for your repo!

I am creating a cube map from my images and use dr.texture(cubemap[None,...], ray_dir[None, ...], filter_mode='linear', boundary_mode='cube') to sample it.

I combine 6 images in order (Right, Left, Top, Bottom, Back, Front) to construct the cube map, which has shapes (6, 256, 256, 3). Then I create the camera rays directions following the OpenGL coordinate system (-Z for FRONT, +X for RIGHT, +Y for UP) to sample the cube map.

I found that this can be sampled to the correct face. For example, the ray_dir pointing to [0,0,-1] obtained pixel from FRONT map. But if I use a direction vector (-0.5,-1,0).normalize() to sample, it actually gets the color that I think should be taken from (+0.5, -1, 0).normalize(), that is, the pixel color on the right part of FRONT texture.

# load cube's 6 faces from png files, store them in a dict
image = torch.tensor(imageio.imread(f'cube/test.png').astype(np.float32) / 255.0) # [256, 256, 3]
cube = {}
for face in ['top', 'bottom', 'front', 'back', 'left', 'right']:
    cube[face] = image

cubemap = torch.zeros((6, 256, 256, 3))

# assign each face to the corresponding position in the cubemap
# in order [Right, Left, Top, Bottom, Back, Front]
cubemap[0] = cube['right']
cubemap[1] = cube['left']
cubemap[2] = cube['top']
cubemap[3] = cube['bottom']
cubemap[4] = cube['back']
cubemap[5] = cube['front']

# -Z for FRONT, +X for RIGHT, +Y for UP
ray_dir = torch.tensor([[0, 0, -1],
                        [-0.5, 0, -1],
                        [0.5, 0, -1]]).cuda()

ray_dir = ray_dir / torch.norm(ray_dir, dim=-1, keepdim=True)
ray_dir = ray_dir.view(1,3,3)
print(f"ray_dir:\n{ray_dir}")

result = dr.texture(cubemap[None,...], ray_dir[None, ...], filter_mode='linear', boundary_mode='cube').squeeze(0)
print(f"result:\n{result}")

it get results:

ray_dir:
tensor([[[ 0.0000,  0.0000, -1.0000],
         [-0.4472,  0.0000, -0.8944],
         [ 0.4472,  0.0000, -0.8944]]], device='cuda:0')
result:
tensor([[[0.5000, 0.5000, 0.5000],
         [0.7510, 0.7510, 0.7510],
         [0.2490, 0.2490, 0.2490]]], device='cuda:0')

the third ray direction points to the front-right direction and should sample brighter pixel from FRONT texture map, but it is dark. What's wrong with my operation?

[s-laine]

My guess is that the input textures are not oriented correctly. As noted in the documentation, nvdiffrast follows OpenGL's convention on texture orientation when setting the cube map contents, see here.

[yifanlu0227]

Oh! So it is not the images looking from the inside out, but following the given convention?