search

ashawkey / stable-dreamfusion

Text-to-3D & Image-to-3D & Mesh Exportation with NeRF + Diffusion.
Apache License 2.0
8.21k stars 721 forks source link

DMTet rendering: alpha and mask usage #197

Open jcao-ai opened 1 year ago

jcao-ai commented 1 year ago

Hi, @ashawkey Be grateful for your great job.

After some trials of reading the code related to the latest DMTet feature. I am a little confused about the alpha/mask usage during DMTet rendering phase.

You first create alpha by interpolating the the ones_like tensor. Won't the result be inflated all 1? https://github.com/ashawkey/stable-dreamfusion/blob/b1cd0fc27f3e887eeddd5d218903ca8ff4bdbdaf/nerf/renderer.py#L857

If I understand correctly, then this alpha tensor is used to generate the mask which is for color masking purpose.

mask = (alpha > 0).view(-1).detach()

# do the lighting here since we have normal from mesh now.
albedo = torch.zeros_like(xyzs, dtype=torch.float32)
if mask.any():
    masked_albedo = self.density(xyzs[mask])['albedo']
    albedo[mask] = masked_albedo.float()
albedo = albedo.view(1, h, w, 3)

Is this masking process redundant or necessary？ Please correct me if I am wrong or if there will be some other potential features in some day. Thanks again.

ashawkey commented 1 year ago

@LitleCarl Hi, the alpha is filled with 1 and 0 (rays intersecting/not-intersecting the mesh). We only need to query those pixels (rays) intersecting with mesh to save computation.