Support for camera_space_light?

[saedrna]

In nvdiffrec, there is a mode for camera_space_light, which is used when an object is placed on a rotating stand table. I noticed that this feature has been removed in nvdiffrecmc. Is it possible to reintroduce this feature without delving into the CUDA/Optix code, and by simply tweaking the light directions in the Python section?

[JHnvidia]

Hi @saedrna,

The best way to solve it would unfortunately be to update the Optix code and add a similar transform to https://github.com/NVlabs/nvdiffrec/blob/main/render/light.py#L98 whenever evaluating / sampling the probe. It requires plumbing through extra data (e.g. the light transform matrix), which is why we dropped it.

A possible, workaround is to rotate the actual light image. In that case, it would should go just before updating the light_pdf https://github.com/NVlabs/nvdiffrecmc/blob/main/train.py#L422 You'd have to:

• start with an empty image in lat-long
• transform each pixel coordinate to a vector using a pytorch version of _tc_to_dir
• transform into original light space, which would be the inverse of the transform used in nvdiffrec
• transform back to lat-long using something equivalent to _dir_to_tc
• fetch luminance from the original light probe image using nvdiffrast's texture function

I'm not sure how well this would work in practice, as it requires re-sampling the light image, which will cause additional blurring.

[saedrna]

We have made a minor modification by adding the rotation matrix (three float3 for each row) and translation vector to the kernel.cu. It seems to work better for our dataset collected on a stand table. However, as you mentioned, the code now looks much less elegant.

[blacksino]

胡教授好，请问可以请教您一些这个repo代码的问题，感觉还是挺复杂的。