bad reconstruction result compared to pytorch3d

[lcc815]

Hi, dear author

I reconstructed a scene by nvdiffrast and pytorch3d, but there are hugh gap between the finnal result (construction algrithm is as the same as cube demo.) pytorch3d result: nvdiffrast result:

I follow the sample to get the rendering function, just like follows:

def __call__(self, pos_ndc, pos_idx, vtx_col, resolution: tuple):
        rast_out, _ = dr.rasterize(self.glctx, pos_ndc, pos_idx, resolution=resolution)
        color, _    = dr.interpolate(vtx_col, rast_out, pos_idx)
        color       = dr.antialias(color, rast_out, pos_ndc, pos_idx)
        return color

Is there any way to improve the reconstruction quality? Or have I made any potential mistakes?

Thanks.

[lcc815]

if look at local, ti's the same. pytorch3d:

nvdiffrast:

one can see there are some strange things in the nvdiffrast.

[s-laine]

This looks similar to the problems in #105. I think it's undersampling, i.e., you're rasterizing the image in too low resolution or too wide crops for the geometry resolution, so that the individual triangles are always smaller than a pixel. The first thing to try would be lowering the model resolution or increasing the rendering resolution.

Are you trying to optimize both geometry and colors, or just the latter? If you're not optimizing geometry, using a texture map would probably work better than using vertex colors — there you can use texture filtering, whereas dense geometry will always alias in too low resolutions.

[lcc815]

I am just trying to optimize only colors, I will try using a texture map just like what you recommended. However, for directly optimizing the vertex colors, I don't think this unexpected results is caused by undersampling. because I have tried to increase image resolution(rasterizer resolution*2, followed by ave_pooling to keep output resolution unchanged), the problem persists. And, if I overfit a single image, I got this result: there is strange white circle arc on the edge correspondingly, the rendering image is also wrong I think In contrast, the result of pytorch3d looks like more reasonable: I think this issue can be easily reproduced.

[s-laine]

Downsampling a rasterized image isn't very effective at solving these kind of issues, because it still allows multiple solutions for reaching the target color, e.g., making some colors brighter and some darker so that they average to the correct shade. Combined with aliasing, I think this is what you're seeing at the periphery.

If you don't have dense enough data to uniquely specify a solution, this is an underconstrained problem, and the only solution would be adding a regularization term that favors smoothness of the solution — this is in fact close to what PyTorch3D's blurring rasterizer does implicitly.

I also note that the geometry doesn't seem to share vertices correctly because there are sharp edges between triangles. This probably also makes the optimization problem a bit more difficult.

[s-laine]

Apologies, I apparently hit a keyboard shortcut that closed the issue. That was not the intent, so reopened.

[lcc815]

OK, got it, thanks for your reply. One more question. Do you think using a texture map will help solve these kind of issues?

[s-laine]

Texturing the mesh (using a filtering texture mode) downsamples the texture adaptively while rendering, so it might alleviate the issue. That said, the solution is still underconstrained if the texture resolution is high enough that multiple texels map to the same pixel all the time. Issues #101 and #105 may contain useful insights. I believe projecting the reference image into texture atlas refers to the technique outlined in issue #1.

[lcc815]

thanks, I will study these issues you mentioned.