Camera position optimization produce wrong result

[Pinnh]

The default background_color of BlendParams is white which cause the optimization to local minima in tutorial camera_position_optimization_with_differentiable_rendering.ipynb

change

blend_params = BlendParams(sigma=1e-4, gamma=1e-4)

to

blend_params = BlendParams(sigma=1e-4, gamma=1e-4, background_color=(0.0,0.0,0.0))

and then add this params to the phong_renderer seems fix this issue

[nikhilaravi]

@Pinnh could you clarify if you resolved the issue? Are you saying that you needed to add background_color=(0.0,0.0,0.0) to the blending parameters in the tutorial?

[Pinnh]

Yes, I resolved，when background_color is not black，although the loss is converge，the render is not correct. It's makes people confusing that in the tutorial the rendered teapot is black background while the running out result is white background.

[Pinnh]

By the way, I found out that the loss of camera_position_optimization is strange when render other meshes, e.g. faces , the loss value is easy to escape the minima and become bigger and never converge, the log is (the array of right side are camera params, the optimal params are near [0,30,300]):

2400 Optimizing (loss 2179.2727) [ -9.63156 25.62216 318.50354] 2500 Optimizing (loss 2154.8718) [ -9.264761 25.362465 318.0325 ] 2600 Optimizing (loss 2134.0764) [ -8.898939 25.122824 317.56253 ] 2700 Optimizing (loss 2110.1489) [ -8.543849 24.897839 317.09262 ] 2800 Optimizing (loss 2091.9775) [ -8.18913 24.690296 316.62326 ] ... 6500 Optimizing (loss 874.5787) [ -0.4055717 33.12555 300.60764 ] 6600 Optimizing (loss 838.6401) [ -0.38454187 33.65727 300.2455 ] 6700 Optimizing (loss 796.5809) [ -0.34294504 34.202816 299.88953 ] 6800 Optimizing (loss 761.1680) [ -0.3351377 34.75815 299.54013 ] 6900 Optimizing (loss 719.7290) [ -0.31356943 35.314827 299.19717 ] 7000 Optimizing (loss 683.2585) [-2.9292086e-01 3.5872055e+01 2.9886130e+02] 7100 Optimizing (loss 641.6945) [-2.6889285e-01 3.6438290e+01 2.9853125e+02] 7200 Optimizing (loss 605.7172) [-2.3675716e-01 3.7002174e+01 2.9821112e+02] 7300 Optimizing (loss 568.4587) [-2.4692412e-01 3.7576008e+01 2.9789758e+02] 7400 Optimizing (loss 534.5746) [-2.1242996e-01 3.8147446e+01 2.9759064e+02] 7500 Optimizing (loss 501.3669) [-1.9008678e-01 3.8709743e+01 2.9729153e+02] 7600 Optimizing (loss 470.4427) [-8.4858187e-02 3.9246456e+01 2.9699301e+02] 7700 Optimizing (loss 451.3885) [2.6072497e-02 3.9738155e+01 2.9669022e+02] 7800 Optimizing (loss 431.7380) [2.2536580e-01 4.0192074e+01 2.9638065e+02] 7900 Optimizing (loss 430.4111) [1.7349118e-01 4.0567482e+01 2.9604160e+02] 8000 Optimizing (loss 441.1376) [ -0.31829834 40.822807 295.65057 ] 8100 Optimizing (loss 460.4149) [ -0.96723837 41.03147 295.22556 ] 8200 Optimizing (loss 499.0586) [ -2.0278277 41.14145 294.75757 ] 8300 Optimizing (loss 550.6298) [ -3.1014965 41.115208 294.21716 ] 8400 Optimizing (loss 618.7897) [ -4.1302342 40.940384 293.60663 ] 8500 Optimizing (loss 691.8552) [ -5.104071 40.643772 292.93915 ] 8600 Optimizing (loss 784.7117) [ -5.97552 40.235306 292.22693 ] 8700 Optimizing (loss 878.1057) [ -6.7887316 39.716953 291.4853 ] 8800 Optimizing (loss 989.1849) [ -7.5157995 39.08833 290.7057 ] 8900 Optimizing (loss 1097.9670) [ -8.091836 38.400185 289.9089 ] 9000 Optimizing (loss 1207.5392) [ -8.720079 37.6598 289.11115 ] 9100 Optimizing (loss 1322.4370) [ -9.2949915 36.919205 288.30798 ] 9200 Optimizing (loss 1428.6519) [ -9.761914 36.16362 287.5138 ] 9300 Optimizing (loss 1535.2325) [-10.323743 35.436043 286.73163 ] 9400 Optimizing (loss 1637.2385) [-10.8353815 34.70769 285.95688 ] 9500 Optimizing (loss 1744.0355) [-11.266405 33.983685 285.19278 ] 9600 Optimizing (loss 1846.3165) [-11.701863 33.285103 284.4479 ] 9700 Optimizing (loss 1943.3331) [-12.119763 32.608543 283.70663 ] 9800 Optimizing (loss 2040.2717) [-12.382797 31.966171 282.98233 ] 9900 Optimizing (loss 2130.0100) [-12.664278 31.324415 282.26852 ]

[KeqiangSun]

Changing the background color in BlendParams doesn't work for me. My method is to change the

(image_ref[..., :3].max(-1) != 0).astype(np.float32)

to

(image_ref[..., :3].max(-1) < image_ref.mean()).astype(np.float32)

In this way the reference image is correct rather than blank. So the key is to obtain the correct model.image_ref.

[gkioxari]

We have now added support for background colors in BlendParams that allows you to change the color of the background. We also fixed the tutorial to use the right background color.

Closing this as the issue seems to be resolved for users! Feel free to re-open if anything new emerges.