Introduce a IPD scale factor

[fernandojsg]

Currently the view/proj matrices as exposed in the API and the spec recommends to use them unaltered. Until now three.js was creating the camera viewMatrix itself, but there's an WIP issue (https://github.com/mrdoob/three.js/issues/10927) to remove that code and use the provided matrices from the WebVR API (https://github.com/mrdoob/three.js/commit/8b31096b86d15975e62ab5d03dcb39fdcce162f8)

My main concern is that we lose the IPD scale factor that we had (https://github.com/mrdoob/three.js/blob/1e85cff6c217e34ae47269cb5750d7e04386e5c9/examples/js/vr/WebVRCamera.js#L110-L111)

I propose to introduce a IPD scale factor so we could still have this functionality to modify the sense of scale depending on our experience. I've been using some tests both in unity and in three.js inspired by the UnrealVR Editor (https://docs.unrealengine.com/latest/INT/Engine/Editor/VR/) and the Tiny Wheels game (http://store.steampowered.com/app/577850/): https://twitter.com/fernandojsg/status/836371728976719872 https://twitter.com/fernandojsg/status/835620186963181568

I understand that the current IPD provided by the API is to provide 1:1 relation between the physical IPD and the virtual IPD, but in some cases like the ones provided before we would like to modify the sense of scale for the user. Another option could be to scale the world itself instead of the camera, but in that case we could screw the physically based shaders, lighting, shadows, physics and so on. So letting the user to introduce a scale factor (1 as default of course) that will be applied to the matrices exposed by the API would solve this.

[ssylvan]

Modifying the IPD itself risky because the view and projection matrices are sometimes very carefully calibrated in IPD-dependent ways. If an app wants to adjust the “scale” of the scene, it seems better to apply a scaling matrix between the world and view matrices. This allows you to effectively scale the world up or down, but without running into the concerns you mention (e.g. physics simulation still happens in unscaled coordinates, and shading is usually done in world or object space, not view space). While the API could provide a mechanism for applying this scale correctly to the view matrices to help here, it seems a bit like an app/engine level concern to me.

[toji]

I can see why this would be convenient in some cases, but it seems like fairly specialized behavior that the average app will not need (and in reality should probably be going out of their way to avoid.) For example, I can't see this working well in mixed or augmented reality scenarios. So I'm reluctant to make it a core feature of the API, especially when apps that have legitimate reasons for wanting this can achieve the same effect with a bit of math.

Still, I'll leave this open for a bit in case one of the other implementers feels strongly otherwise.

[kearwood]

I had a conversation with fernandojsg in back-channel. This raised some points in support of an IPD scale factor:

• Changing IPD is not the same as scaling (ie. the near and far planes change with scaling)
• Without another option, devs may try to decompose the matrices to get the information they need, and do it themselves. This leads to bad results on devices they can't test on
• Could ask the API if changing IPD is possible for the current device, and provide a way to set an IPD scaling option. Some devices (such as AR headsets) can refuse to allow IPD scaling.

[toji]

Scaling the view matrices should not have any effect on the projection matrices, so the near and far planes don't have to change. Furthermore scaling the IPD independent of the head translation is just flat out wrong, especially if the intent is to make the user feel bigger or smaller.

[kearwood]

It's true that scaling the view matrix won't affect the near and far planes directly; however, the net effect is that the near and far planes -- in world units -- is effectively different.

This side-effect may be desirable in many circumstances, as when you are simulating a physical shrinking in size, you may want to get closer to objects.

Perhaps what may be more general-purpose would be an option to set the near and far plane. This would be useful outside of the case of scaling IPD. Of course the range of near and far plane values could be constrained by the devices as necessary.

[toji]

We already have that in the form of depthNear/depthFar, unless I'm misunderstanding something? (And for the record the way those are exposed continues to feel a bit awkward to me. If there's a better way to handle them, especially for cases like this, I'm all ears.)

[kearwood]

I suppose the depthNear and depthFar values could be multiplied by the inverse of the scaling factor to keep them the same in world coordinate space after the scaling.

[brianchirls]

In my experience, scaling the IPD will very quickly make one sick and is not something we should encourage.

There are ways to scale the world, but they raise other considerations. For example, when scaling the world, you might make part of the scene semi-transparent on top of a stable floor/grid or mask part of the view the way Google Earth VR does. Or, you might do a fade out and fade in as a transition. Also, if you're going to adjust scale, you'll want to keep the controllers the proper size. In that case, it's best to put the bulk of your scene under a parent object that scales while keeping the controllers and other "meta" objects at their original scale.

[ssylvan]

Is there a good use case for when you might actually want IPD scaling? We tend to strongly encourage developers to keep things as "real" as possible to avoid comfort issues. E.g. if you want something to "look bigger", it should be because it actually is bigger, and everything in the app, like physics and shading, treats it as if it were that bigger size. We should avoid having the spec encourage something that is usually an anti-pattern unless there are some great use cases.

[fernandojsg]

@brianchirls I believe you're talking about the experience itself, if you scale something very fast you could become sick the same way as if you rotate/translate something really fast, not about the scale transformation itself. right? Otherwise just have a look at the examples I posted like Tiny Wheels or Unity/Unreal VR editors.

@ssylvan I agree that for experiences where, for example you're a little bug and keep running around the forest, it's preferred to pre-scale the assets for the whole game and just use the standard IPD. But for experiences where you've dynamic scale, lets say:

• Alice in wonderland style
• An educational experience where you could travel around the body and become smaller or bigger to traverse some pieces of it.
• An experience where you can choose different prehistoric animals and see how they see each other in scale and move around the scene at different sizes.
• A supermario VR? That modifies it size if you have or not a mushroom? :)
• VR-Editor such as the commented before

I agree that in AR it doesn't seems to make sense to have a IPDScale <> 1 but VR I see it quite useful.

What I would like to have, ideally, is a way to achieve this kind of effects without decompose the matrix, apply translation, generate a new one, generate a new combined frustum (In case this issue is accepted for the API), open to possible errors by the developer.

Honestly I'm not sure if we could have a simpler way to do that directly in the js app/engine without adding that scale factor, I should do the maths but let's say:

• Set depthFar/Near to 1/IPDScale
• Read the left/rightViewMatrix that should use the updated depthFar/Near
• Scale the left/rightViewMatrix byIPDScale` factor
• Read the combined frustum (If it's implemented finally)
• Scale the combined frustum Is there something I've been missing maybe?

[kearwood]

@brianchirls Agree on changing scale quickly; however, there are opportunities to change it very slowly as a gameplay element, or even changing it instantaneously. These approaches should not give the same sickness effect.

[spite]

While it's interesting, I think this is trying to solve a problem that should be addressed by the 3D engines: there should be a way to properly scale the scene and not mess with the shading, physics, etc. calculations. Speed of change to not provoke motion sickness, etc. should of course also be addressed by the engine/developer.

Scaling the IPD, besides adding complexity to the API by providing a not-necessarily tweakable parameter, wouldn't address the fact that the controllers should remain the correct size, for instance.

[ssylvan]

I agree that this smells like something an engine or other middleware would do. I'd also like to re-emphasize a point @toji mentioned earlier - scaling the IPD is not correct and will make people sick if you don't also scale all translational motion as well (motion parallax cues would not match the stereo cues). So a feature like this would have to scale everything, not just IPD, in order to work with 6DOF head motion (even fake 6DOF head motion, from neck models).

Also, as @brianchirls points out there are many app-specific choices one might want to make here (e.g. does the sword I'm holding scale too, or does it stay the same size while the rest of the world scales?), so it just seems like we couldn't even do anything automatic here even if we wanted to, since the app still has to make some choices about what gets scaled and what doesn't.

[kearwood]

I suppose the key question, is whether the algorithm described in @fernandojsg 's last comment should be implemented in UA or if it should be documented as an approach using the currently defined API interface.

I am mixed on this, but believe it's a productive discussion.

Cons: Additional surface area to the API that currently could be done in Javascript engine code. Pros: If this algorithm is not appropriate for any future hardware devices, the UA could implement the correct algorithm for those devices without updating sites.

Perhaps some other terminology could be used other than "IPD scaling" to better describe the effect, while giving the UA flexibility to implement the right effect based on the hardware and user application.

[fernandojsg]

@spite I agree that features like speed change, and scale the scene using while taking care of the physics, shading and so on is something that engine/dev should address. I don't mean that you'll always want to modify the perception of the scale by adjusting the IPD, sometimes you could scale the scene, it really depends on the use case and you need to take into account problems like resolution while moving to very big/small scales. I was following the same way Unity does, there you could just modify the scale of the camera and it will scale the IPD, so you could just use that or scale the scene itself.

Regarding the @ssylvan and @brianchirls comments about the sword/controllers scale vs the whole scene, you could always have a cameraRig including your controllers and your camera, and you'll scale the whole cameraRig, it will scale the controllers, the head position and the IPD itself so you'll have the desired effect. (That's the way I did on the previous examples).

So as this is something major engines like Unity offer and it's being used by production titles, I could imagine that people could eventually want to have something similar in the browser. What if for example we want to export Tiny Wheels to WebVR/WebGL/WebAssembly, we'll hit a incompatibility here that will require us to modify a big part of the code probably.

[ssylvan]

@fernandojsg Are you suggesting that WebVR gains some notion of a transform hierarchy a la Unity so that the app can choose which items to scale? Because if so, that seems like a pretty high level feature to add to an API like this, and I would be inclined to point people to middleware such as ThreeJS or BabylonJS instead for those kinds of game engine features.

[toji]

I don't feel like comparisons between WebVR and Unreal/Unity are productive. One is an API, the others full game engines. Comparisons between WebVR<=>OpenVR and Three.js<=>Unity are probably more informative.

It's worth noting that none of the hardware APIs expose such scaling either, so Unity and co. are handling it all at the engine level, which is exactly what I've been advocating for.

[kearwood]

@ssylvan The scaling is a visual and culling effect only, and shouldn't affect physics or scale only certain objects. The physics and selective scaling would be application specific and done with the kind of rig @fernandojsg describes above.

[kearwood]

We need to qualify if this needs to be done at the API level, or if it can be done in content with the algorithms described earlier in this thread.

If we leave this up to the game engine / three.js / WebAssembly code to apply, perhaps we should just document this as an example. Perhaps we need a direct way to ask if such scaling is a "good idea" for particular hardware or not. (ie. AR and C.A.V.E. systems probably wouldn't get the desired effect if the scaling was applied by content)

[fernandojsg]

I completely understand your concerns about webvr vs engine, but I didn't mean to mix them both maybe I didn't express myself correctly. In fact until now we had the IPD Scale factor implemented in three.js core, by applying the camera' x-scale to the IPD while composing the per eye matrix: https://github.com/mrdoob/three.js/blob/b4ab886a18b34f454a83811d2f946934a9f964f7/examples/js/vr/WebVRCamera.js#L110-L111 I wanted to expose that it's something that most commonly used engines like unity or unreal exposes to their users so we would like to have the same in js engines like three.js. The problem is that to achieve the same effect in three.js with the latest changes, as we're going to start using WebVR API's matrices, is to have a specific check if the camera's scale <> 1 so instead of using the WebVR API's matrices as is, we should decompose them and create a new ones with the scale applied. The same will go for Unity/Unreal exporters as if the user is applying scale changes to the camera, it won't be possible to use the WebVR API matrices but generate the new ones as we did before in three.js.

[fernandojsg]

I believe the discussion shouldn't be around if it's better to scale the scene itself or the camera, as we've seen that there use cases for both, but about if we're providing enough developer ergonomics to do that in both cases.

[ssylvan]

@kearwood I think these things are going to depend on the experience, we can't make blanket assumptions here. E.g. did I get hit by the shrink ray, or did my house get hit by the magnifier ray? Depending on the case the app would want things to behave very differently. So I think there's enough app-specific choices to make on these things that we couldn't actually do a good job with anything automatic at a platform API level (we just don't have enough context about what the app wants).

@fernandojsg I haven't looked at the rest of your code, but I suspect that what you're currently doing is actually wrong on a 6DOF device. Simply scaling the IPD is not equivalent to adjust the "scale" of the world, and will make people sick because head translation remains unscaled. Imagine an extreme case where you make the IPD 10x bigger, then when you're standing still the world will appear 10x closer as far as the stereo cue is concerned. However, as you translate the head you actually get 10x less translation than your brain expects, leading to motion parallax/stereo vision mismatch and discomfort. The correct thing to do is to scale all world coordinates, not just the IPD of the camera.

[toji]

to achieve the same effect in three.js with the... WebVR API's matrices... we should decompose them and create a new ones with the scale applied

I think I've missed something. Why would this necessitate decomposing the matrix?

[fernandojsg]

@ssylvan the way to work on these engines, and the one I reproduce in three.js is to create a camera rig that includes the controllers and the camera, and when you want to scale it you scale the whole camera rig, so you're scaling correctly the controllers, the camera translation and the IPD as expected so it will feel completely natural.

@toji Sorry, I didn't mean saying "decomposing" the matrix itself but modify it instead of using it directly. Basically I meant to follow the approach described in my previous comment:

What I would like to have, ideally, is a way to achieve this kind of effects without decompose the matrix, apply translation, generate a new one, generate a new combined frustum (In case this issue is accepted for the API), open to possible errors by the developer.

Honestly I'm not sure if we could have a simpler way to do that directly in the js app/engine without adding that scale factor, I should do the maths but let's say:

• Set depthFar/Near to 1/IPDScale
• Read the left/rightViewMatrix that should use the updated depthFar/Near
• Scale the left/rightViewMatrix byIPDScale` factor
• Read the combined frustum (If it's implemented finally)
• Scale the combined frustum Is there something I've been missing maybe?

[ssylvan]

@fernandojsg Ah okay, that does make sense, you're grabbing the scale from the camera rig. However, I still think there's a bug. The offset between two eyes it not necessarily one-dimensional, so you have to scale it along all three axes (not just scale.x). Assuming that you allow the scale to be non-uniform here. In particular, manufacturing tolerances can lead to a small vertical offset which is very important to get right for comfort since human eyes don't independently move vertically a lot (very small vertical disparities can lead to headaches, we very carefully calibrate these offsets so things will look right).

If you just scale everything with a scaling matrix (either before doing physics/gameplay if that's what you want, or wedge a scaling matrix in-between the world and view matrices if you want it to be an entirely visual effect) then things will work correctly, and you won't have do any modifications or decompositions of the matrices.

EDIT: Actually, since the use case you have is scaling the camera itself what you actually want in these instances it to apply a (possibly non-uniform) scale to the view matrix, right? So just grab the matrices from the API, then apply your scale to them and use those combined matrices as your view matrices. No need to compute offsets or scale the IPD especially (or even have a separate scaling matrix), just scale the view space. This seem like it would be trivial with our current API and not need any special accommodation, no?

[toji]

Feels to me like the general consensus here is that this is something that should be left to the app. Closing to make it easier to focus on other outstanding issues.

[fernandojsg]

@ssylvan @toji yep, I've been doing some experiments and I think it won't be hard to keep that in the engine without need to screw the matrices that we get from the API, I just need to check how the near/far are affected. Thanks for the productive feedback!

[toji]

No problem! And thanks for the discussion, even if it didn't ultimately lead to any spec changes.