low mesh quality with 256 tet + 128 spp

[iraj465]

Hi, Fantastic piece of work, unreal to imagine this works so well out of the box. Appreciate this great work guys!

I'm getting good "smooth" quality meshes withe dmtet_grid:128 but i wanted to increase the details and hence generate the centred 256 tets and set dmtet_grid:256 keeping other hyperparams constant, but getting far worse quality with 256 dmtet. Why is that? Is there a technical problem that i'm not seeing here?

dmtet_grid:128 sample

dmtet_grid:256sample

Config:

{
  "ref_mesh": "/test_1",
  "random_textures": true,
  "iter": 2000,
  "save_interval": 100,
  "texture_res": [
    2048,
    2048
  ],
  "train_res": [
    1080,
    1920
  ],
  "batch": 3,
  "learning_rate": [
    0.03,
    0.003
  ],
  "dmtet_grid": 256,
  "mesh_scale": 2.1,
  "validate": true,
  "n_samples": 12,
  "denoiser": "bilateral",
  "display": [
    {
      "latlong": true
    },
    {
      "bsdf": "kd"
    },
    {
      "bsdf": "ks"
    },
    {
      "bsdf": "normal"
    }
  ],
  "background": "white",
  "out_dir": "nerf_materials",
  "ks_min": [
    0,
    0.1,
    0
  ],
  "ks_max": [
    0,
    1,
    1
  ]
}

[JHnvidia]

Hi @iraj465,

Thanks for checking out the code. We've also noted that training becomes increasingly difficult as tessellation increases. The problem is essentially that geometry gradients only propagate through silhouette edges. As the DMTet resolution goes up, the random noise initialization makes the gradients too noisy. It's the same in the original nvdiffrec codebase.

Unfortunately, I think it will be hard to get to work by only tuning the parameters. We haven't really looked into increasing detail, but some "off the top of my head" implementation alternatives would be:

• Using a low frequency, e.g. MLP, encoding of the SDF prior to running DMTet.
• Running DMTet on a coarser resolution first and using the result to initialize the fine resolution grid (instead of starting from noise)
• Filtering / blurring the DMTet gradients by some kernel that is large atthe beginning of the training process successively decreases.