search

microsoft / MixedRealityToolkit-Unity

This repository is for the legacy Mixed Reality Toolkit (MRTK) v2. For the latest version of the MRTK please visit https://github.com/MixedRealityToolkit/MixedRealityToolkit-Unity
https://aka.ms/mrtkdocs
MIT License
6k stars 2.12k forks source link

Integration with Unity's UI Toolkit #10239

Closed agzimmerman closed 9 months ago

agzimmerman commented 3 years ago

Describe the problem

Recently UI Toolkit (UnityEngine.UIElements) provided the functionality needed to render its runtime GUIs on 3D objects in the world space and to interact with that GUI using a mouse. The features are available in their latest preview packages which work with the Unity 2020.3 LTS but also are now in the core of Unity 2021.2 beta.

Unfortunately this seems to only work with a mouse, not with any MRTK pointer. I have tested the gaze pointer and hand ray using input simulation as well as rays from VR controllers when compiled for the Oculus Quest.

Describe the solution you'd like

MRTK pointers should be able to interact with UI Toolkit visual elements.

Describe alternatives you've considered

Right now we have to develop two GUIs, one using UI Tookit and one using UGUI, with only the UGUI one working with MRTK.

I tried testing this with Unity 2021.2 beta but there are some known bugs with MRTK on that version which I think are slated for MRTK 2.8.

I suspect it may be a bug on the UI Toolkit side where their current implementation only checks for mouse events and ignores other types of interactions. But when I look into this, I get lost in a sea of different event systems, input systems, and input modules; so I am hoping that someone here has already wrapped their head around this and can point me in the right direction. I am optimistic that this will just work once MRTK 3 works with the next (2021) Unity LTS; but my team could greatly benefit from having this functionality right now.

Additional context

Here is a recorded example (from UI Tookit's 3DRenderTexture example) using Using MRTK 2.7.2, Unity 2020.3.18f1 (LTS), and UI Toolkit 1.0.0-preview.17:

https://user-images.githubusercontent.com/9367569/134925858-7c6ac666-2825-4e5a-afa3-1b1382a24cbc.mp4

I can provide a lot more information if there is someone interested in getting MRTK and UI Toolkit working together.

JakobAnarkyLabs commented 2 years ago

+1 on this request. Since the UI Toolkit / UI Elements seems to be the future way of building UI in Unity, it would be important for MRTK to support and embrace that. UGUI is now somewhat supported by MRTK, but it seems like a small 'niche' which is not really encouraged. It would be nice, if MRTK could integrate with the UI Toolkit more wholeheartedly.

agzimmerman commented 1 year ago

@JakobAnarkyLabs or anyone interested: At my company we figured out how to use the same UI Toolkit (UI Elements) GUI in screen space and in world space with MRTK 2 interaction and the internal solution has been working well for a while now. For now we published a XR Interaction Toolkit (oh which MRTK 3 is based) version in the Unity Asset Store as FlexXR: UI Elements in World Space in Mixed Reality. It would be much nicer if this was handled by Unity or MRTK at some point but I hope this low price (but admittedly paid) asset can help some people.

IssueSyncBot commented 9 months ago

We appreciate your feedback and thank you for reporting this issue.

Microsoft Mixed Reality Toolkit version 2 (MRTK2) is currently in limited support. This means that Microsoft is only fixing high priority security issues. Unfortunately, this issue does not meet the necessary priority and will be closed. If you strongly feel that this issue deserves more attention, please open a new issue and explain why it is important.

Microsoft recommends that all new HoloLens 2 Unity applications use MRTK3 instead of MRTK2.

Please note that MRTK3 was released in August 2023. It features an all-new architecture for developing rich mixed reality experiences and has a minimum requirement of Unity 2021.3 LTS. For more information about MRTK3, please visit https://www.mixedrealitytoolkit.org.

Thank you for your continued support of the Mixed Reality Toolkit!