Image-space derivatives for environment illumination

[RatTac]

Thanks for sharing this fantastic piece of software!

I have a small question regarding the immage space derivatives:

Let's suppose I want to do environment mapping (as in the env_phong example). I would first compute the reflection vectors per vertex, then interpolate them, finally use the texturing function with the interpolated reflection vectors and the "cube" sampling option. For passing the image-space derivatives "uv_da" I would use the secondary output of the interpolation function. So far, so good...

However, what happens if I would like to have an additional normal map for my object? Then I cannot interpolate the reflection vectors directly but I would need to interpolate the view vectors instead, compute the normals per pixels (via texturing using the normal map) and then do the calculation of the reflection per pixels to get pixel-wise reflection vectors.

However, when samping the environment map using these pixel-wise reflection vectors, how would I then specify the uv_da parameter?

[s-laine]

Hi @RatTac — it's great seeing people doing more advanced stuff with nvdiffrast, so I'm happy to help. I don't know how exactly your normal map is structured, so here's a general explanation.

You will first need to precalculate derivatives for your normal map, i.e., change with respect to u and v. This could be as simple as taking central differences in horizontal and vertical direction, or something more advanced.

Then, as you say, you interpolate (u, v) and the view vectors into pixels, fetch the normal map at (u, v), and compute the reflection vector according to the interpolated view vector. In addition, you need to fetch the precalculated normal map derivatives w.r.t. u and v at (u, v), and convert these to normal map derivatives w.r.t. pixel X and Y via chain rule, i.e., multiplying by the 2x2 Jacobian matrix formed by the differential outputs of the interpolation operation for u and v. Differentiating the formula for calculating the reflection vector w.r.t. X and Y should now yield the reflection vector derivatives w.r.t. X and Y (based on the just computed normal map derivatives w.r.t. X, Y and the view vector derivatives w.r.t. X, Y that you get from the differential output of the interpolation operator) which is what you need to do a filtered fetch from the cube map.

Hope this helps you forward. For a deeper dive into the topic, the standard reference for chasing differentials along rays is Igehy's 1999 paper Tracing Ray Differentials.

[RatTac]

Yes, in my opinion nvdiffrast is so good as it provides only the basic modules required for many tasks and thus has a very high flexibility. Thanks for sharing it! :smile:

Thanks also for the remarks regarding my question. Let me summarize, if I understood correctly.

Normal_Map: I have a 512x512 normal texture atlas in tangent space. This I would probably first need to convert to object space, right? UVs: I would interpolate the uvs to get the uvs per pixel UVs_DXY: This is generated using the interpolation function from above View: I would interpolate the view vectors to get the view vectors per pixel View_DXY: This is generated using the interpolation function from above Normals: I would texturize using UVs and Normal_Map to get the normals per pixel Reflections: I would compute the reflection vectors using View and Normals using the reflection equation Normal_Map_Duv: I would use central differences to compute the derivatives of the normals according to u and v Normals_Duv: I would texturize using UVs and Normal_Map_Duv to get the derivatives of the normals according to u and v per pixel Normals_DXY: This should be computable as you said via the chain rule as Normals_Duv * UVs_DXY

I now end up with View_DXY and Normals_DXY. You said that "Differentiating the formula for calculating the reflection vector w.r.t. X and Y should now yield the reflection vector derivatives w.r.t. X and Y". Could you elaborate on this? As reflection = reflect(view, normal) I would need to calculate the formulars for reflect_dview and reflect_dnormal, right? But now I get a little bit lost on how to proceed for calculation Reflections_DXY (what I want)... :cry:

[s-laine]

Yes, this is exactly right. Having the normal map in object space would indeed be a great way to simplify things, as long as you don't need the object to deform.

In the last step you need to differentiate the reflection formula w.r.t. pixel coordinates, not w.r.t. the arguments. If my math is correct (you should double-check this!), given the reflection formula

r = v - 2 * dot(v, n) * n

the pixel derivatives would be

drdX = dvdX - 2 * ((dot(dvdX, n) + dot(v, dndX)) * n + dot(v, n) * dndX)
drdY = dvdY - 2 * ((dot(dvdY, n) + dot(v, dndY)) * n + dot(v, n) * dndY)

where every variable is a 3D vector, and drd{X,Y}, dvd{X,Y}, and dnd{X,Y} are the derivatives of reflection, view, and normal vector w.r.t. pixel coordinates, in that order.

[RatTac]

All right, now it finally made click in my head 😄 Thanks a lot for your reply, really appreciate your work!