Lighting estimation for WebXR

[NellWaliczek]

There's already an API on the web for Lighting Estimation (https://w3c.github.io/ambient-light/) but it currently only supports getting the ambient intensity. It would be useful for improving the realism of AR rendering to have slightly more information than that. For example the color temperature would be helpful and is generally available on the underlying XR platforms that would back WebXR. There are also other more advanced lighting options available on some platforms that would be worth investigating as optional features. One example is this from ARKit: https://developer.apple.com/documentation/arkit/arenvironmentprobeanchor

[blairmacintyre]

I've seen a bunch of discussion about that API being removed (I think some browsers already have?). The issue (as with the devicemotion API) is that it can be accessed in the background without permission, and there were some crazy threat scenarios painted for how it can be abused.

[NellWaliczek]

To clarify, I wasn't necessarily proposing using the ambient light api! Just pointing out that we needed to pay attention to it's existence. Do you have a link to the discussion about it being avoided?

[blairmacintyre]

Ok, digging back in, it seems mostly a firefox dev mailing list discussion (e.g., here https://groups.google.com/forum/#!topic/mozilla.dev.platform/DcSi_wLG4fc/discussion and here https://support.mozilla.org/en-US/questions/1222676), and it's unclear where they are on their thinking here.

That said, when I read through the spec (https://w3c.github.io/ambient-light/), it feels like some of the concerns may be relevant if we enable this capability at the "lowest" level of permission. For example, a malicious script could potentially detect that multiple users are together by the ambient light sensing pattern, or even if they were in a certain place. These are the kind of threats that we need to be conscious of.

Now, if we only enable estimation when users have granted access to "world knowledge" of some form, I think many of these threats are less compelling, since there is so much other information to leverage.

[kyungtae]

I want to implement the "Lighting estimation for WebXR" for my contents, but IMHO, the current AmbientLightSensor API seems not suitable for AR.

The LightEstimation of the AR platforms are tightly coupled with the "frame", because we can estimate the light of the AR scene with the background (camera preview ) of the scene, not with the additional sensor like illuminance sensor.

So, why don't we make separated API for WebXR, that are related to the XRFrame ? (Light estimation for AR #401)

[cwilso]

@kyungtae I think that was the rough conclusion (that we probably needed a separate API from the Ambient Light API), for the reason you stated (tied with frame).

[kyungtae]

IMHO, the LightEstimation API could be like below.

Because we can estimate the light of the AR scene with the environment (camera preview ) of the scene, we can add "XRLightEstimation" attribute in XRFrame.

[SecureContext, Exposed=Window] interface XRFrame {
...
  readonly attribute XRLightEstimation? lightEstimation;
}

Additionally, the XRLightEstimation consists of the intensity and the color, and the Application can apply them into the intensity and the color of the ambient light. Color would be optional because some platforms may support only the intensity.

[SecureContext, Exposed=Window] interface XRLightEstimation {
  readonly attribute double intensity;
  readonly attribute double? color;
};

[blairmacintyre]

When this came up before, the question raised was "are such simple values adequate"? Will platforms want to support this simple approach? Do we need to support a couple of variations or kinds of data?

I would be in favor of having this be a separate repo with a proposal that could be worked on, and it would be good if some of the platform engineers could chime in.

[toji]

I think the simplicity of this proposal works as long as we specify that it's an ambient estimate. So if the above IDL is tweaked to say

[SecureContext, Exposed=Window] interface XRLightEstimation {
  readonly attribute double ambientIntensity;
  readonly attribute double? ambientColor;
};

I think that implicitly makes the lighting estimation object more extensible in the future. I personally have questions about whether or not we'd want to have the color be a single double (in which case it should probably be ambientColorTemperature) or a full RGB value. I know that Google and Apple at least differ on this, and I'm not fully sure the reason why or what the benefits are one way or the other. @cwilso has said in the past that lighting temp is apparently a well known concept in photography, so I'm sure that played a part in Apple's design.

[TrevorFSmith]

I received feedback that it's better to keep proposal Issues in here for search purposes, so I'm moving them back until we can think through how to handle it.

Ada, when you make the README for the repo, please add a link to this Issue.

[klausw]

@TrevorFSmith Did you close this issue intentionally? I think it might be helpful for searchability to keep it open until it is converted to a README or other representation for this spec component.

@toji about ambient color vs color temperature, I think an RGB ambient color seems strictly more powerful. A color temperature represents black body radiation that can be converted to RGB, i.e. via lookup tables or a calculation, but the inverse isn't really possible. For example, if the scene is lit with an unnatural light color such as green or purple, that can be represented as an RGB color, but color temperature is restricted to a one-dimensional red-to-white-to-blue path through the color space, so forcing it into that representation would potentially lose data.

[TrevorFSmith]

@klausw Yes, once we create a feature repo all conversation should happen there. So, I recommend creating a new Issue over there and if necessary refer back to this Issue.

[kearwood]

May I suggest that SH (Spherical Harmonics) be considered as part of the format to express the environment lighting in a directional dependent manner?

https://en.wikipedia.org/wiki/Spherical_harmonics

This has the advantage of being small (feasible to calculate and store hundreds on a mobile device), scalable (can easily access various levels of detail), and used ubiquitously in game engines for this purpose.

[kearwood]

For background on use in lighting specifically: https://en.wikipedia.org/wiki/Spherical_harmonic_lighting

[kearwood]

Even more specifically, we could use PRT (Precomputed Radiance Transfer):

https://en.wikipedia.org/wiki/Precomputed_Radiance_Transfer

It should be possible to express a simple ambient / color temperature value in this format as well as extract such a value from PRT data that includes the higher frequency domain.