Undefined behaviour of light (and audio) under viewer scale

[hybridherbst]

A growing usecase of glTF is the use for e-commerce products and lightweight interactive experiences, especially on mobile and with Augmented Reality in mind.

While working in this area for many projects, we found that currently, some behaviours are entirely undefined in the glTF spec and thus all viewers either do it obviously wrong (because it's nowhere mentioned) or have differing behaviour.

Ideally, there would be a non-normative section in the glTF spec (or similar) that talks about behaviour under viewer scale, encourages extensions where that matters to explain how they assume to work, and thus serves as a rough guiding principle. I don't think this would need to be normative right now, but I imagine that in the future with more extensions that range into interactivity and spatial cases, this is becoming more of an issue.

---

Consider the typical scenario of placing a model in AR with model-viewer, and the model has spatial audio. Should scaling the model down to 10% of the size

• assume that the model is now small, but the distance to the viewer is approximately the same?
• assume that the viewer is now large and the model hasn't changed scale, and thus the distance to the viewer is now much larger?

Unfortunately such considerations were omitted from lighting in glTF, and thus lighting in AR is often broken right now.

To explain why that's not trivial, here's my take:

• lighting effects should work as if the viewer got bigger (if I scale a lamp with a spot light down, I don't want the light to become brighter, it should look the same just smaller)
• audio volume should work as if the model became smaller (if I scale an AR radio down, I want to hear it at the same loudness)
• audio spatialization should work as if the viewer got bigger (if I have 3 spread out emitters in a space and scale that space down, I still want clear spatial separation between these sources)

---

Here's some examples from

• https://github.com/google/model-viewer/issues/2474

https://user-images.githubusercontent.com/2693840/121874453-a87a8900-cd07-11eb-9191-7b7ad9bf30b9.mp4

Note that when placing the model in AR, the effect is correct; but when scaling the model, the appearance is that the lights get brighter and closer to each other! That's not what you want when you're scaling down a little architecture model of a city with some street lights!

And a live problematic file that shows that artistic intent gets immediately lost upon viewer scale

• https://prefrontalcortex.de/labs/model-viewer/upload/Giant-Isopod-Animation/

References

• KHR_audio, which will have the same problem
• https://github.com/KhronosGroup/glTF/pull/2137#issuecomment-1109075382

[hybridherbst]

@DrX3D I think with more and more AR and VR usecases where glTF is used, it would be great to get some guidance on this topic. Just delegating to implementors to individually figure it out has proven problematic, as essentially all I heard is "it's undefined in glTF, why should we care". There could at least be a note ackowledging the problem and suggesting what solutions might be.