Support Spherical Harmonics Lighting

[pomelyu]

🚀 Feature

Add a new class SphericalHarmonicsLights to the Lights family. There is an issue(https://github.com/facebookresearch/pytorch3d/issues/349) related to this, but it has been closed, and no one tried to implement it.

Motivation

Spherical Harmonics Lights are frequently used in the analysis-by-synthesize method for 3D face reconstruction. There are several papers use this way, for example DECA.

Pitch

I had already implemented a draft for this feature. I applied the first 9 SH parameters from Basel illumination Prior on the BFM model. Here are some results. I am not sure the results are correct or not and not know how to verify them. Further, the scale of the coefficients might not have a standard. I will take time to check the above problems.

[patricklabatut]

Thanks for the suggestion and for sharing some information on your use case!

Environment lighting is definitely on our radar as a potential future feature. More specifically, diffuse irradiance implemented with spherical harmonics (i.e. following the classical [1][2]) is indeed a likely first candidate for this.

[patricklabatut]

[...] I am not sure the results are correct or not and not know how to verify them

Ensuring the computations are correct with documented conventions to facilitate interoperability while maintaining compatibility with the existing direct lighting calculations will be key there.

In my experience, correctness is best ensured with end-to-end knowledge of the computations and conventions. With the specific source of data at hands (the Basel Illumination Prior you are using), one would need to know in depth how the irradiance SH coefficients have actually been generated. Then rendering correctness can be partially checked by reviewing (visually and numerically) results on simpler geometry (i.e. a sphere) for each SH base function and also by comparing results to reference renderers with established conventions if needed.

[patricklabatut]

Further, the scale of the coefficients might not have a standard

AFAICR a factor of 1 / π may or may not be pre-applied to irradiance maps / irradiance SH coefficients (to normalize over the considered hemisphere) depending on conventions and / or shading model. This again requires being familiar with how the input data for rendering is being generated.

[pomelyu]

Hi @patricklabatut,

Thanks for your kindly response. I totally forget I could use sphere to verify the result. After doing that, I found the scale is not correct. I will study more about SH lights and figure out the actual formula used in the Basel Illumination Prior. By the way, does there already have any plans to support SH light?

[patricklabatut]

By the way, does there already have any plans to support SH light?

Yes, as mentioned in the above comment, this is definitely on our radar for future releases.

[pomelyu]

I fix the problem finally, and get the some result as described in Basel Illumination Prior. If needed, I will create the PR. Thanks

[jonilaserson]

Hi, I found this thread for the same reason @pomelyu did. I'm not yet familiar with how lighting works in graphics. However, been reading a lot of face-reconstruction papers recently, and many of them use the same model of spherical harmonics, with 3 bands and 9 tunable parameters per channel. (for example here)

@pomelyu, is this still on your todo list for a PR?

[pomelyu]

@jonilaserson you could see the code here

[jonilaserson]

@pomelyu Beautiful! Thank you. Do you mind sharing a link as a reference for the computation you did?

[pomelyu]

The formula could be found in "Lambertian reflectance and linear subspaces". I also refer to the other two repositories(DPR, DECA) and follow the order of parameters as Basel Illumination Prior.