Camera pose optimization with RGB channel instead of alpha channel

[ykzzyk]

Hello! Thanks for sharing the great work.

🐛 Bugs / Unexpected behaviors

I have a question regarding the camera pose optimization using RGB channel instead of alpha channel. It appears to me that when using RGB channel to perform the MSE loss function, the optimization process does not make sense.

Instructions To Reproduce the Issue:

When I follow the tutorial and makes some changes:

1. Tested with the Cow mesh instead of teapot mesh
2. Updated the pose to be a little far away from the camera
3. Changed the silhouette shader to softphong shader
4. Used RGB channel to perform the optimizing process, the result looks this:

The starting position and target position:

The optimizing process

I wonder why when I use RGB channel to minimize the errors, but the errors are increasing over time? To prevent the gradient explode, I also clip the grad norm, but it does not seem to help much.

[bottler]

This is a modelling question which we cannot help with.