search

microsoft / MixedRealityToolkit-Unity

This repository is for the legacy Mixed Reality Toolkit (MRTK) v2. For the latest version of the MRTK please visit https://github.com/MixedRealityToolkit/MixedRealityToolkit-Unity
https://aka.ms/mrtkdocs
MIT License
6.01k stars 2.12k forks source link

Unity UI buttons have priority over MRTK buttons #3322

Closed cellarmation closed 5 years ago

cellarmation commented 5 years ago

Overview

A Unity UI button being behind a MRTK/HTK button causes user interactions (gaze/highlight and clicks) to be received by the Unity UI button preventing the MRTK/HTK button from being usable. It seems like the Unity UI controls have a higher priority and will steal interaction events unless the MRTK/HTK button is significantly in front.

This worked as I would expect in Unity 2017.2.0f3 & HoloToolkit-Unity-2017.4.2.0, but it does not in Unity 2017.4.15f1 & HoloToolkit-Unity-2017.4.3.0, so I believe something has changed in Unity or MRTK.

Expected Behavior

The first button to be hit with a raycast from the user's gaze/controller should have priority and get the event independent of if it is a Unity UI button or a MRTK/HTK button, like it does in Unity 2017.2.0f3 & HoloToolkit-Unity-2017.4.2.0.

Actual Behavior

Unity UI button's normally get the event unless the MRTK/HTK button is significantly in front of the Unity UI button The attached image shows the camera gazing at the timer button that has been moved in front of the Unity UI button in the example InputManagerTest scene. As you can see the Unity UI button is highlighted (yellow).

unityui

Steps to reproduce

1) Create a new project 2) Import HoloToolkit-Unity-2017.4.3.0.unitypackage 3) Import HoloToolkit-Unity-Examples-2017.4.3.0.unitypackage 4) Change the scene to HoloToolkit-Examples\Input\Scenes\InputManagerTest.unity 5) Move the "Timer Button" in front of the "Unity UI Button" 6) Run the project and gaze at the timer button

Unity Editor Version

Unity 2017.4.15f1

Mixed Reality Toolkit Release Version

HoloToolkit-Unity-2017.4.3.0

cellarmation commented 5 years ago

After doing a bit of further testing I have found that the issue isn't present in Unity 2017.2.0f3 with HoloToolkit-Unity-2017.4.3.0, but is in Unity 2017.4.15f1 HoloToolkit-Unity-2017.4.2.0. So, the change in behavior has been caused by Unity doing something different between Unity 2017.2.0f3 and Unity 2017.4.15f1.

cellarmation commented 5 years ago

The issue is also not in Unity2017.3.1f1 (with HoloToolkit-Unity-2017.4.3.0) so it appears its a Unity 2017.4 thing.

cellarmation commented 5 years ago

This is still an issue as of 2017.4.19f1.

Mixed Reality Toolkit does two independent ray-casts in FocusManager::UpdatePointer(). One is against physics objects, RaycastPhysics(), and one against the UI, RaycastUnityUI(). Inside RaycastUnityUI there is logic that allows the UI raycast to overwrite the physics ray-cast if it meets certain conditions, including the distance of the raycast hit being shorter.

Testing pre and post Unity 2017.4 for the case I am interested in (Unity buttons behind MRTK buttons stealing interactions) I can see the difference is due to the bool "overridePhysicsRaycast" being set when it wasn't before. This is set in RaycastUnityUIStep() due to the distance being reported by the UI ray cast being shorter than it was in earlier versions, the difference is enough that when compared against the MRTK raycast it seems that the UI button must be in front of the MRTK button. The underlying API call that is returning different results is eventSystem.RaycastAll() in EventSystemExtensions::Raycast().

I am very interested to know if anyone has any insight to this behavior and if there are any potential workarounds or fixes. The whole EventSystemExtensions class seems to be there to work around limitations with the UnityUI raycast behavior, but I don't really understand it.

cellarmation commented 5 years ago

So I have worked out the issue first appears in Unity 2017.4.3f1. Looking at the release notes I suspect it is due to this fix: "(934908) - UI: Fixed Canvas raycast start position for screen and world space camera modes."

It seems that the nearClipPlane of the MixedRealityCamera and the UIRaycastCamera now have an impact on the distances reported by the raycasting. I suspect the comparison of the UnityUI and Physics raycasts is currently broken as it doesn't take into account the near clip planes correctly.

Tweaking the values to make the nearClipPlane of the UIRaycastCamera less than the nearClipPlane of the MixedRealityCamera can bring back the original behavior. However, I have not found any particually good values as it also appears to effect the depth of the cursor when over objects and the ability for the cursor to be able to hit UnityUI controls.

cellarmation commented 5 years ago

Fixing issue #3359 and being aware of the fixed Cursor Surface Distance that intentionally offsets the cursor from the surface accounts for why I couldn't find a fix just tweaking the logic of the two ray casts. I have explained a working fix for both issues in #3359.

Yoyozilla commented 5 years ago

Looks like relevant PRs are merged

Please reopen if it still repros with the latest Unity 2018 + MRTK release