sapo17 / BachelorThesis

Simple material parameter estimator tool using Mitsuba 3 and Dr.Jit.
2 stars 0 forks source link

Apply smoothing to a mesh #52

Closed sapo17 closed 1 year ago

sapo17 commented 1 year ago

Apply smoothing during vertex_position optimization -- maybe not each iteration, but if loss exceeds the margin

sapo17 commented 1 year ago

Notes from kaolin, Optimizing a mesh using a Differentiable Renderer tutorial:

During training we will use different losses:

- an image loss: an L1 loss based on RGB image.
- a mask loss: an Intersection over Union (IoU) of the segmentation mask with the soft_mask output by DIB-R rasterizer.
- a laplacian loss: to avoid deformation that are too strong.
- a flat loss: to keep a smooth surface and avoid faces intersecting.

For that we need to compute the laplacian matrix and some adjacency information (the face idx of faces connected to each edge)

Currently, not sure whether we can use mask loss. But other losses could be implemented in our tool.

Mentioned tutorial further shows:

...
### Compute Losses ###
image_loss = torch.mean(torch.abs(image - gt_image))
mask_loss = kal.metrics.render.mask_iou(soft_mask,
                                                gt_mask.squeeze(-1)) # ?

# laplacian loss
vertices_mov = vertices - vertices_init
vertices_mov_laplacian = torch.matmul(vertices_laplacian_matrix, vertices_mov)
laplacian_loss = torch.mean(vertices_mov_laplacian ** 2) * nb_vertices * 3

loss = (
          image_loss * image_weight +
          mask_loss * mask_weight +
          laplacian_loss * laplacian_weight
      )

### Update the mesh ###
loss.backward()
vertices_optim.step()
texture_optim.step()
sapo17 commented 1 year ago

Regularization loss incoming for both vertex pos. optimization (Large Steps in Inverse Rendering of Geometry Loubet et al 21), and grid volume opt. (influenced Vicini et al. 22, SDF Optimization)

sapo17 commented 1 year ago

See branch #61 and commit.

See AdvancedVertexOptimizer. Still a lot of room for improvement, which will be handled in another ticket/issue.