depth map is not changing under different camera position

[Crane-YU]

Hi @s-laine, I was rendering the depth map under different camera positions, which means for each camera position I would render the depth map correspondingly. However, I found that the depth value does not change under different camera positions and the depth values always match with the ones rendered from the first camera position. Is this a desired outcome?

[s-laine]

How are you rendering the depth map? If you're placing depth in a vertex attribute and interpolating, you of course need to compute these separately for each camera position.

[Crane-YU]

Hi @s-laine, I followed nvdiffrec to use DepthPeeler to take vertex positions on clip space, and I recalculated the vertex positions on clip space for each different camera position. The thing is that although I computed the DepthPeeler for each different camera position, my depth map seems incorrect.

[Crane-YU]

the initial camera position is at the front (depth map shown below) when I rotate the camera to the back, the rendered depth image is shown as follows: which seems incorrect to me.

[s-laine]

You will also have to recalculate the vertex depth attribute that you are interpolating to get the final result.

Or is this the z/w buffer from the rasterizer? It's a bit difficult to help as I don't know what exactly your code is doing.

[Crane-YU]

Thank you @s-laine , I indeed forgot to update the vertex depth attribute after each new camera.

[SlimeVRX]

Hi @Crane-YU

I want to get depth from mesh, Can you send me a sample code? Thank you very much!

[SlimeVRX]

Many thanks!

This is my sample code

import imageio  # Importing imageio library, used for reading and writing image data
import numpy as np  # Importing numpy library, which is commonly used for large, multi-dimensional arrays and matrices, along with mathematical operators on these arrays
import torch  # Importing PyTorch library, a popular library for machine learning research
import nvdiffrast.torch as dr  # Importing nvdiffrast's PyTorch interface, a library used for rasterization, mainly in graphics rendering

# Define a helper function that creates PyTorch tensors on GPU ('cuda' device)
def tensor(*args, **kwargs):
    return torch.tensor(*args, device='cuda', **kwargs)

# Initialize a CUDA context for rasterization
glctx = dr.RasterizeCudaContext()

# Define the position of vertices in 3D space. Each vertex has x, y, z coordinates and w (for homogenous coordinates)
pos = tensor([[[-0.8, -0.8, 1, 1], [0.8, -0.8, -1, 1], [-0.8, 0.8, -1, 1], [0.8, 0.8, 1, 1], [0.4, 0.4, 1, 1], [0.3, 0.3, 1, 1], [0.2, 0.2, 1, 1]]], dtype=torch.float32)

# Get the z (depth) values from the position tensor
depth_ = pos[..., 2:3]

# Convert the depth tensor to have only one z-value per vertex
depth = tensor([[[(z_val/1)] for z_val in depth_.squeeze()]], dtype=torch.float32)

# Define the triangles by referring to the indexes of vertices in 'pos' tensor
tri = tensor([[0, 1, 2], [2, 3, 1]], dtype=torch.int32)

# Rasterize the triangles to generate pixel coverage and depth (z-buffer)
rast, _ = dr.rasterize(glctx, pos, tri, resolution=[256, 256])

# Interpolate depth values across the rasterized pixels
out, _ = dr.interpolate(depth, rast, tri)

# Compute old minimum and maximum depth values
old_min = torch.min(out)
old_max = torch.max(out)

# Define new minimum and maximum depth values (for normalization)
new_min = 0
new_max = 255

# Normalize the output tensor to range [new_min, new_max]
out = (out - old_min) / (old_max - old_min) * (new_max - new_min) + new_min

# Convert the output tensor to a numpy array, repeat it across color channels, and convert it to uint8 type
out = np.squeeze(out.cpu().numpy())
out = np.repeat(out[:, :, np.newaxis], 3, axis=2).astype(np.uint8)

# Notify the user that the image is being saved
print("Saving to 'tri.png'.")

# Save the rasterized and interpolated depth values as an image
imageio.imsave('tri_.png', out)