can we store depth image for splatfracto

[sumanttyagi]

after 3d reconstruction how do i store the depth images for individual images ?

[maturk]

use ns-render and set rendered_output_names = depth

[sumanttyagi]

as i can see the code , will it use colmap to store the depth images ? or is nerf or splatfracto value will be stored for depth ?

[sumanttyagi]

@maturk if i am able to convert it to grayscale, can we get absolute depth ? also what convention should i follow as when i am converting to any grayscale image the depth is not projecting right , the images produce from this blue means near and red means farthest ?

[sumanttyagi]

@maturk depth images are not so smooth it having some spread splats

[maturk]

use the 'raw-depth' option in the render script. colmap is not used here. The depth comes from the splatfacto model (projection of the gaussians and alpha blending). I think using 'depth' applies a color map but 'raw-depth' will save the raw depth. The depth images are not necessarily supposed to be smooth... this is just the output of splatfacto depth channel. It has no depth supervision so it is what is is.

[sumanttyagi]

@maturk i did the same saved the raw -depth values , and then projected it with 0 255 normalized value using below script

import numpy as np
import gzip
# Load the compressed numpy array
with gzip.open(r"\raw-depth\frame_00125.npy.gz", 'rb') as f:
    image_array = np.load(f)

# Now you can use the 'image_array ' variable as a regular numpy array
print(image_array )
normalized_array = ((image_array - image_array.min()) / (image_array.max() - image_array.min())) * 255

# Convert to uint8 datatype
normalized_array = normalized_array.astype(np.uint8)
cv2.imwrite('output_normalized255_125_image.png', normalized_array)

i am ending up with the output as below :

1. 

if i export the same scene as depth images below output is coming

2. 

if i convert 2 to grayscale value directly using below script

import cv2

# Load the image
image = cv2.imread(r"\depth\frame_00125.jpg")

# Convert the image to grayscale
gray_image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
cv2.imwrite('output_gray_image125.png', gray_image)

3.

• [ ] i think raw and this is actually same as we can see 2 and 3 look same the difference in pixel value is same ?
• [ ] do you think the depth images generated heatmap is having some convention ? what color spectrum/colour space represents what or simply converting to to grayscale will give output
• [ ] can we create depth images for some different pose ?
• [ ] (most important) while visualization using three.js i can see the cursor values getting changes on the surface i.e. there is depth changes however same is not coming while using ns_render for creating the depth images
• [ ] can we set the camera position to acquire the depth images ?

[maturk]

please check the code for the colormaps: https://github.com/nerfstudio-project/nerfstudio/blob/main/nerfstudio/utils/colormaps.py. They are just colormaps that are useful for visualizing boolean or scalar values by mapping them to a synthetic RGB frame that human eyes can visualize... cv2 has these as well e.g. cv2.applyColorMap(img, cv2.COLORMAP_INFERNO) etc...

yes you can render rgb/depth from any pose you want. You need to look into the render scripts more. its all in the code I am not familiar with three.js and I dont understand what that has to do with this. ns_render either saves an image or a video... not your cursor values.