search

NVlabs / nvdiffrec

Official code for the CVPR 2022 (oral) paper "Extracting Triangular 3D Models, Materials, and Lighting From Images".
Other
2.09k stars 222 forks source link

Jaggy geometry and texture details getting washed out #62

Closed bsespede closed 1 year ago

bsespede commented 1 year ago

Hello, first of all thanks for such a great work! I was briefly experimenting with the algorithm and I'm having some issues.

1) As you can see in the images the geometry looks very jaggy. I've created a higher res tet grid (256 - as big as my GPU would fit) but it seems to still have random "holes" and "spikes". Is there something I can do to prevent this? My scene has 108 frames and reconstruction results look very good on COLMAP. 2) Texture details seem washed out in comparison to the results in the paper. I thought it might be due to my extrinsics/intrinsics but they have rather low reproj error (~2pixels) so I'm really surprised (I've even extended the projection matrix to use different focal length and principal point per axis). Is there any setting I can tweak to improve this or insight why this might be happening? I've tried increasing iterations even further but it didnt help

val_000107_normal val_000107_opt val_000107_ref

Kind regards, Braulio

JHnvidia commented 1 year ago

Hi,

It's always quite tricky to pinpoint sources of quality drop. The main sources for geometric errors we've found are srtong speculars, inconsistent/varying lighting, and camera pose estimation. Unfortunately I'm not sure exactly how much error should be expacted at 2 px of reprojection error as we didn't focus much on camera poses.

It's possible to regularize (low pass filter) geometry through regularization, e.g. by using an MLP to encode the DMTet SDF. While we haven't released this code, a quick test could be to reduce the tet-grid resolution and see if the geometry improves a bit.

There's some additional information in this issue: https://github.com/NVlabs/nvdiffrec/issues/37

In recent follow up work we noted that it's sometimes beneficial with a normal smoothness loss (which can be implemented as a screen space loss term): https://arxiv.org/pdf/2206.03380.pdf

tlok666 commented 1 year ago

Hi,

It's always quite tricky to pinpoint sources of quality drop. The main sources for geometric errors we've found are srtong speculars, inconsistent/varying lighting, and camera pose estimation. Unfortunately I'm not sure exactly how much error should be expacted at 2 px of reprojection error as we didn't focus much on camera poses.

It's possible to regularize (low pass filter) geometry through regularization, e.g. by using an MLP to encode the DMTet SDF. While we haven't released this code, a quick test could be to reduce the tet-grid resolution and see if the geometry improves a bit.

There's some additional information in this issue: #37

In recent follow up work we noted that it's sometimes beneficial with a normal smoothness loss (which can be implemented as a screen space loss term): https://arxiv.org/pdf/2206.03380.pdf

Thanks for your advice. Would you release code of the mentioned paper(https://arxiv.org/pdf/2206.03380.pdf)?