search

andrewhou1 / GeomConsistentFR

Official Code for Face Relighting with Geometrically Consistent Shadows (CVPR 2022)
https://openaccess.thecvf.com/content/CVPR2022/html/Hou_Face_Relighting_With_Geometrically_Consistent_Shadows_CVPR_2022_paper.html
MIT License
114 stars 17 forks source link

Differentiable ray tracing? #1

Closed RodneyPerhaps closed 2 years ago

RodneyPerhaps commented 2 years ago

Amazing job there! After browsing through the training code, I can't find the logic for the ray tracing part, how is this part implemented?

image

This part of the code seems to have something to do with lighting, but I don't see how this is related to ray tracing.

andrewhou1 commented 2 years ago

Hi there, thanks for your interest in our work!

So the raytracing code comes after this (lines 371-522), but let's first discuss the snippet of code that you included here. If I can refer you to Eq. 4 in our paper, the idea is that regions under hard shadow should have intensities that match the ambient light in the environment and regions that are not under hard shadow should be fully illuminated (ambient component+directional component). That is why we define the last two variables "ambient_light" and "full_shading". Once we compute the shadow mask below this code, any hard shadow regions will have intensities close to "ambient_light" and any illuminated regions will have intensities close to "full_shading".

Now to discuss the actual raytracing component. Again, this is below the code that you've pasted here. What we do is for each point A in the depth map, we sample 160 points from the depth map between A and the light source L. For each of the sampled points P_i, we compute the distance from P_i to the light ray AL by using ALxAP_i as discussed in the paper. Then, we take the P_i with the smallest distance and feed that minimum distance to Eqn. 8 in the paper to compute the shadow mask value. This smallest distance is referred to as d_min in the paper. Intuitively, this is a way to differentiably compute a shadow mask value and estimate the locations of hard shadows. If d_min is small, that means there is some sampled point that closely intersects with the light ray, which means that the original point A is blocked by some surface and under hard shadow. Please let me know if anything else is still unclear!

RodneyPerhaps commented 2 years ago

Hello Andrew! Thank you for your very detailed answer! This explains a lot!

zhipeng-fan commented 1 year ago

Hello Andrew! Thanks for the detailed illustration. I was wondering since we are taking the min distance here, will that create some discontinuities and therefore making it no longer differentiable?

andrewhou1 commented 1 year ago

Hi there! So the min function is differentiable with respect to values but not differentiable with respect to the index. In this case, since the min distance is outputting some value each time, the gradient will always be able to backpropagate through that value since it represents the computed distance for a point. If instead we were trying to output an index i for the minimum distance point and use that in further steps, then the differentiability would be broken.

zhipeng-fan commented 1 year ago

I see. Great explanation! Thank you!