search

facebookresearch / pytorch3d

PyTorch3D is FAIR's library of reusable components for deep learning with 3D data
https://pytorch3d.org/
Other
8.81k stars 1.32k forks source link

I meet problems in compute the gradient of UV texture map when use the differentiable render #1626

Closed godfatherwang2 closed 1 year ago

godfatherwang2 commented 1 year ago

❓ Questions on how to use PyTorch3D

Hello, I have some question about the Differentiable rendering，I set the raw texture UV map as parameter to be optimized, and load the mesh. However when I render the mesh , I find when using the output image to compute the loss. The gradient of the texture UV map is zero. I use the code below：

uvmaps = Variable(torch.randn(1, 1024, 1024,3).cuda(), requires_grad=True) mesh = Meshes(verts=[param3d_middle], faces=[f_uv],textures=TexturesUV(maps=uvmaps,faces_uvs=[f_uv], verts_uvs=[uv_coords])) image = renderer_textured(mesh,R = R,T = T)[...,:3] loss_rgb = ((image-target_image)**2).mean() loss_rgb.backward() print(uvmaps.grad)

bottler commented 1 year ago

What is renderer_textured?

godfatherwang2 commented 1 year ago

Thank you for your attention，I use the below code to create this oject. Perhaps I need a sample code to learn how to optimize the UV map with pytorch3D. camera = FoVPerspectiveCameras(device="cuda") renderer_textured = MeshRenderer( rasterizer=MeshRasterizer( cameras=camera, raster_settings=raster_settings_soft ), shader=SoftPhongShader(device="cuda", cameras=camera, ) )

What is renderer_textured? Thank you for your attention，I use the below code to create this oject. Perhaps I need a sample code to learn how to optimize the UV map with pytorch3D. camera = FoVPerspectiveCameras(device="cuda") renderer_textured = MeshRenderer( rasterizer=MeshRasterizer( cameras=camera, raster_settings=raster_settings_soft ), shader=SoftPhongShader(device="cuda", cameras=camera, ) )

bottler commented 1 year ago

When you visualise the output, do you see the mesh?

godfatherwang2 commented 1 year ago

Yes, I am sure I can see the rendered mesh. Is it possiable that I use the wrong rasterizer or shader ?

bottler commented 1 year ago

The rasterizer does not matter. The shader is fine. I don't understand what is happening here. Can you paste the image of the output of the renderer here? Does it contain many different colors (showing that it is affected by many elements of the map)?

godfatherwang2 commented 1 year ago

I'm very sorry, I think I made a very basic mistake. I found the gradient to be 0 because of an issue with my IDE's display format. I think PyTorch3D is a great project.