what's correct way to construct 3D mesh ?

[graysuit]

For ZoeDepth, we use: gradio_im_to_3d.py huggingface demo

What I tried:

import torch
import numpy as np
from PIL import Image
from unidepth.models import UniDepthV1, UniDepthV2
import trimesh
from zoedepth.utils.geometry import depth_to_points, create_triangles

model = UniDepthV2.from_pretrained("lpiccinelli/unidepth-v2-vitl14") # or "lpiccinelli/unidepth-v1-cnvnxtl" for the ConvNext backbone

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device)

rgb = torch.from_numpy(np.array(Image.open('1500.jpg'))).permute(2, 0, 1) # C, H, W
predictions = model.infer(rgb)

depth = predictions["depth"]
xyz = predictions["points"]
intrinsics = predictions["intrinsics"]

def depth_edges_mask(depth):
    depth_dx, depth_dy = np.gradient(depth)
    depth_grad = np.sqrt(depth_dx ** 2 + depth_dy ** 2)
    mask = depth_grad > 0.05
    return mask

depth = depth.squeeze().cpu().numpy()

pts3d = depth_to_points(depth[None]).reshape(-1, 3)
Z = pts3d[:, 2]
ZMin = Z.min()
ZMax = Z.max()

# scale = 50
# pts3d[:, 2] *= (scale / 100.0)
triangles = create_triangles(1500,1500, mask=~depth_edges_mask(depth))
mesh = trimesh.Trimesh(vertices=pts3d, faces=triangles, vertex_colors=rgb)
file = "new.stl"
mesh.export(file)

Result:

But result looks to be having blocks. Also some holes inside compared to zoedepth. Was i'm doing wrong ?

[lpiccinelli-eth]

TBH, I have not tried it, but I think it may be related to the interpolation used in the pre- and post-processing. In particular, we use "nearest-exact" interpolation by default to upsample to the original shape. You can try setting it to bilinear and let me know!

To set it to bilinear add this line: model.interpolation_mode = "bilinear", before calling model.infer.

[graysuit]

@lpiccinelli-eth Hi Thanks for replying I really appreciate that.

I tried bilinear, now looks better:

But there's another issue Z axis wise some mesh looks separate:

Original image:

Maybe I'm doing some mistake while plotting ?

[lpiccinelli-eth]

Thank you for your answer.

I do not know exactly the reason for this, but in the V2 model, we noticed a bug with some GT that creates artifacts in the bottom corners of the images, as if the depth there is close to zero, we are currently retraining our models, including smaller to solve that problem, by Monday we should be able to have the new models and UniDepthV2 corrected.