Prism Effect: Improve Implementation in VR

[bastianilso]

Current Prism Effect does not make people confused about their pointing when they leave VR.

[bastianilso]

• Instead of only shifting the controller, we could try implementing this by performing a rotation of the camera.
• having an arm representation might also provide a stronger effect.

the reasoning is that the visual target doesnt change and hence it is easier to repeatedly get the same target with or without the current controller shift.

[bastianilso]

Here is a quick first implementation of the prism effect creates the shift by camera rotation.

Todo:

• Test whether this implementation induces the prism effect.
• Implement UI as a slider. (fixed)
• Test the logging and support it in WAM patterns.
• Craete visual representation of an arm.

[larsevald]

Ideally, we should test the prismatic after-effect in healthy controls with different implementations

[bastianilso]

Some interesting thoughts from Kessler Foundation on implementing a prism-like effect in VR:

"Participants move their hand more and more to the left while the VR environment showed their hand reaching straight ahead."

[bastianilso]

@Xav1204 if you get time, this issue will require some research:

1) I would like if you borrow the prism glasses in my office and make a technical comparison of analog prism effect, versus VR prism effect (as currently implemented). and write some suggestions for what we could do to make the VR prism effect better.

2) You may also investigate the possibilities for a visual representation of an arm: for example, does SteamVR provide something we can use? maybe newer versions do? or maybe there is a different standardized asset from the XR framework?

please report back in this issue what you find out on point 1 and 2 (just as plain text).

[Xav1204]

@bastianilso

1. For the first idea that we've thought with lucas. We could create two other cameras ( one for left side and one for the right side ) and make a rotation and an offset of each camera to have a better renderer because when we wear glasses, if the prism effect is on the left side, the renderer on the right eye is the same that without glasses. We have just a little rotation on the left side for this eye. And for the left eye, we have a rotation and an offset. So it's why we've thought to create two others cameras that have the half of the main camera size with a little offset and rotation. So when we want to activate the prism effect, we activate the prism effect camera and we make a rotation of the main camera.
2. For the arm prefab, there don't have free asset that exists and don't know if they are suitable with steamVR. The only thing that we could do is to take the hand model .fbx and modeling an arm.

The idea is hard to explain so if you have problem to understand it, please tell me. I will come for making you a drawing. It would be better to understand the idea.

[bastianilso]

@Xav1204 thanks for the test. Well, I think generally speaking the idea of introducing both rotation + offset, could make sense. However, I doubt that we can find theoretical ground for why we would need to introduce different amount of rotation/offset on each eye. The size and angle of the prisms are the same on both left and right eye, so looking out of either of them should yield the same effect.

A recent paper suggests that 20-30 degrees rotation gave the most deviation in users. "Moreover, the finding that only participants adapted to 30-degrees showed biased bisection performance suggests that a critical threshold must be reached in order to induce significant visuomotor transfer." (Gammeri et al, 2018)

So I think that the next step for us could be something like this: Try implementing a 20 degree rotation in a scene with 2 objects placed at a 30cm from one another (0.3m in unity units). Take pictures of how this scene is rendered inside each eye of the VR glasses (I believe there is also API functionality screenshot the imagery sent to the glasses from Unity). Then make a corresponding physical setup in the real world and use the prism glasses, which should be 20-degree prisms (there is a number '20' marked on each prism). Then take pictures of the result in the lens. This way you can compare how objects shift in the virtual world to the physical world.

If it's easier for you, you can also start simply trying these two implementations in comparison to one another.

After we try this, we can then try to introduce some offset too, if that can help getting it closer to the real-world effect. It could be that the camera needs to be offset slightly backwards/forwards. sideways could also work as another type of approximation. Feel free to try them all. We can make additional changes to the angle, fx making it smaller/larger.

by the way, in that paper they also render a complete arm in VR, see figure: https://www.tandfonline.com/doi/full/10.1080/09602011.2018.1502672#_i7

[Xav1204]

@bastianilso I've made some draw and screens to show you the problem that I have with the implementation of the prism effect. You can see in the following draw the theory of the prism effect that I've simulated in the real world. When we look the mole 11 and that we aim it, if we put the prism effect glasses, the hand and the pointer don't move. The only thing that change is that our view focus the mole 9. In the following draw, you can see the result that we have on unity. Indeed, the rotation that we apply on the camera replicate the prism effect of the real world and the cursor on the mole is still focused on the mole that we targeted before to activate the prism effect. However, when the rotation is made, the hand move according to the rotation that we've applied on the camera.

No Prism Effect :

Prism Effect :

[bastianilso]

Thanks @Xav1204, this clarified things for me a lot, I think I understand the problem now. I have some work questions for you.

• does the hand actually move? (e.g. the XYZ coordinates of the hand change when camera rotates? or is this an illusion, so to speak)

• if it does, why does it? (is it because it is a child of the game object we rotate?)

• is there a way to prevent it? if not, could we use e.g. a) controller offset in the inverse direction or b) apply inverse 20 degree rotation which only affect the hand?

• does activating the prism effect in terms of function calls, modify anything else other than the camera object?

[Xav1204]

@bastianilso I've made the test.

1. XYZ hand coordinates don't change when we rotate the camera. It's like an illusion
2. If we use an offset or a rotation for the controller, we won't have the right result. Indeed, if we use an offset, the cursor will move too and with a rotation, it doesn't work. It isn't the right result that we want.
3. The only thing that we rotate is the child PrismEffect in the camera rig object.

[Xav1204]

@bastianilso As you can see, when we want to disable the prism effect, we need to change the value with the slider until 0. However, if the therapist want to do this quickly, he could surpass this value and it could be upset for the patient to see the camera moving slowly.

So, if we want to keep the slider to permit to the therapist to change the prism effect step by step to not upset his patient. I think that it could be a good thing if the therapist can change the prism effect to the min and max value directly and if he can disable the prism effect quickly. So I think we can add three buttons under the slider. Like this, the therapist can choose how he want to activate the prism effect.

Moreover, as the glasses have a 20° effect, put -20 and 20 in the slider should be more logical. Please, let me know what are you thinking about these ideas.

[bastianilso]

FYI, first iteration of the improved prism effect was merged as part of pull request #246

[bastianilso]

We discussed whether to do the following for prism effect, though yet to be completely decided:

• Cursor and controller should be invisible, no hover effects or triggering of moles in this state.
• We could then measure when people press the trigger button and "point directly" at the moles. we would then need to log how the direct angle (pointing) corresponds to a position on the wall, to measure how close they are to the target.

Lars/Iris will think further how to think this into the treatment program setup.