search

MixedRealityToolkit / MixedRealityToolkit-Unity

This repository holds the third generation of the Mixed Reality Toolkit for Unity. The latest version of the MRTK can be found here.
BSD 3-Clause "New" or "Revised" License
355 stars 89 forks source link

MRTK support for Apple VisionOs and Vision Pro #88

Open IssueSyncBot opened 11 months ago

IssueSyncBot commented 11 months ago

Original issue opened by:

@njannink @njannink

With the upcomming release of Apple VisionOs what will be the vision of MRTK for supporting this and the Apple Vision Pro.

Would love some insights to be able to form our architecture roadmap.

ISSUE MIGRATION

Issue migrated from: https://github.com/microsoft/MixedRealityToolkit-Unity/issues/11611

IssueSyncBot commented 11 months ago

Original comment by:

@camnewnham camnewnham

Based on the packages used some thoughts/discussion points:

For early testing, you can apply to Unity's beta program: https://create.unity.com/spatial

Disclaimer: Not a core MRTK contributor

IssueSyncBot commented 11 months ago

Original comment by:

@AMollis AMollis

Thank you @njannink , at this time we have no plans to support Apple Vision Pro. However, community contributions are welcome, and we encourage the community to add this support.

yosun commented 6 months ago

@njannink i think that MRTK UI components for VisionOS would at least give wider reach. (and longer extended project lifetime for those of us who have spent time with mrtk).. unity polyspatial does not have UI support for VisionOS other than a prefab button... whereas MRTK has so many of these necessary spatial computing prefabs already built

@AMollis @IssueSyncBot

yosun commented 6 months ago

@camnewnham

njannink commented 6 months ago

We are moving away from MRTK to be able to support the VisionPro and at the same time dropping support for the Hololens. After Microsoft announced that Hololens3 isn't going to happen anytime soon and they ditched MRTK it became clear that we should also leave this behind us.

camnewnham commented 6 months ago

It looks like things should work in immersive mode... I saw a video somewhere of the simulator running the MRTK dev templates. Passthrough seems to be more difficult though, a big challenge being that code-based shaders are unsupported.

faysalfdt commented 6 months ago

Honestly it would be super helpful to have MRTK shader support fixed for Polyspatial, there's nothing essentially ready UX wise and requires a lot of rework and porting to make stuff work

ptc-rgrasset commented 6 months ago

We are also interested about this feature. Anyone from the community is actually trying to get that working ? (we don't want to duplicate effort here).

yosun commented 6 months ago

@ptc-rgrasset are you guys going to update Vuforia for VisionOS?

ptc-rgrasset commented 6 months ago

I can't and won't make any comment for Vuforia or PTC about VisionOS.

For now I'm mainly interested to get feedback, status, or any progress from anyone who has tried to build MRTK 3.0 (or 2.8) for VisionOS.

yosun commented 6 months ago

@ptc-rgrasset Ah sorry - They closed down vuforia forums and no one seems to respond to the stackoverflow, so the off chance that you could properly direct the question.

ptc-pscheper commented 6 months ago

Hey @yosun, if you're interested in seeing Vuforia Engine for VisionOS, feel free to share your device feedback via our GitHub issue-only repository: https://github.com/PTCInc/vuforia-engine/! 😄

keveleigh commented 5 months ago

I'd say I'm personally happy to help make any required changes to get MRTK up and running, within my capacity! I don't anticipate having access to a device, and the simulator seems to be Mac-only (please let me know if I'm mistaken!), so I'd be relying on everyone here to report issues and whatnot.

But PRs and specific requests are always welcomed!

keveleigh commented 5 months ago

Looking through some of the existing reports in this thread:

I suggest MRTK3 Input package should migrate/support Unity's XR Hands package.

support for this was added in #504, though it's not yet the default in the MRTK profile we ship.

Honestly it would be super helpful to have MRTK shader support fixed for Polyspatial

This would be a https://github.com/microsoft/MixedReality-GraphicsTools-Unity request (CC @Cameron-Micka)

given that minimal work is required to get an MRTK3 app running on ios/android with ARFoundation I expect this to also be mostly out of the box

I would assume this as well, and it's a big reason we migrated to having ARFoundation as a dependency in MRTK3 instead of re-doing it all ourselves again like we did in MRTK2. But please let us know if you run into issues!

unnanego commented 5 months ago

We got two devices, so I will try it out next week and post here if I don't forget

yosun commented 5 months ago

Hey @yosun, if you're interested in seeing Vuforia Engine for VisionOS, feel free to share your device feedback via our GitHub issue-only repository: https://github.com/PTCInc/vuforia-engine/! 😄

Posted already actually quite a while ago - https://github.com/PTCInc/vuforia-engine/issues/15

Will re-post? There should be an AI to automate this already...

jelmer3000 commented 5 months ago

Maybe interesting for the people here, I just started trying to get things to work on VisionOS without too much planning in advance. So far, simple input works (click buttons, drag objects, play that piano) and I've done some work on the shaders: made new shader graphs for the more common shaders (from the scene below), with all properties from the original shaders (so that it's easy to swap materials). But by far not all properties' functions are implemented in the shader graphs. Also while the MRTK shaders are uber shaders that compile based on flags, I just made one bug big shader so performance might suffer eventually.

Anyway feel free to give it a try and contribute if you like. https://github.com/jelmer3000/MixedRealityToolkit-Unity-PolySpatial

Next steps:

mrtk3-on-visionos

Khalid2188 commented 5 months ago

I managed to make MRTK3 work in fully immersive mode here are steps

1- Download the sample in Apple visionOS XR Plugin and understand the Main scene 2- Disable both hands interactors under MRTK3 XR rig 3- Use gaze interactor from MRTK3 and configure it by adding correct XRI Actions following sample project 4- It should work well in Unity, Simulator and Vision Pro

https://github.com/MixedRealityToolkit/MixedRealityToolkit-Unity/assets/37955941/02b43d59-caac-4601-9152-dce0fee619d4

Cameron-Micka commented 5 months ago

I'm happy to fix any issues with Graphics Tools and Apple Vision Pro in immersive mode. Once I get a device, I can do some testing and sanity checks.

Supporting spatial apps is a bit harder since there are a lot more shader requirements. Eventually I'd like to have better Shader Graph support for that.

jbienzss commented 4 months ago

@jelmer3000 I've been taking a look at your fork and it's promising. Thank you for starting that work. I did notice that it's currently 18 commits behind MRTK/main and I was curious if you'll have time to keep your fork up to date?

I was at VisionDevCamp last week. 100 hackers, 50+ Vision Pro's all in one place. It was epic.

https://www.linkedin.com/feed/hashtag/?keywords=visiondevcamp

There was a lot of interest in getting XR Hands working. Me and two others spent most of the hack trying to get XR Hands working in Mixed Reality Unbounded (exclusive) mode. We could get ray-based interactions to work but not grab manipulations. The moment we switched to Immersive mode it started working. We even had XRI 3.0 with Near-Far interactor on the rig and we still couldn't get MR to work. I'd love to see brilliant MRTK minds like @Cameron-Micka and @keveleigh help get things working.

@Cameron-Micka, what would it take to make a case for your improved shader graph support? I notice that issue has been open for nearly 2 years. As you're probably already aware, on visionOS in Mixed Reality mode (bounded or unbounded) Unity does not actually perform the rendering. Rendering is handed off to Reality Kit and therefore custom HLSL shaders are not supported in this mode. Shaders written with Shader Graph are translated by Unity as best they can in order to fit into Reality Kit. The only way any version of Graphics Tools could possibly work in MR on AVP would be if it's written based on Shader Graph. Here's the documentation on which Shader Graph features are currently supported on visionOS:

https://docs.unity3d.com/Packages/com.unity.polyspatial.visionos@0.1/manual/ShaderGraph.html

Cameron-Micka commented 4 months ago

@Cameron-Micka, what would it take to make a case for your improved shader graph support? I notice that issue has been open for nearly 2 years. As you're probably already aware, on visionOS in Mixed Reality mode (bounded or unbounded) Unity does not actually perform the rendering. Rendering is handed off to Reality Kit and therefore custom HLSL shaders are not supported in this mode. Shaders written with Shader Graph are translated by Unity as best they can in order to fit into Reality Kit. The only way any version of Graphics Tools could possibly work in MR on AVP would be if it's written based on Shader Graph. Here's the documentation on which Shader Graph features are currently supported on visionOS:

https://docs.unity3d.com/Packages/com.unity.polyspatial.visionos@0.1/manual/ShaderGraph.html

Unfortunately since this isn't currently a requirement of my current work responsibilities, I don't see having time to do this in the short term. But it's certainly something I would like to do!

yosun commented 4 months ago

@Cameron-Micka what are your work responsibilities? Since MS discontinued Hololens, this is the future of spatial computing and MR/XR. There is a lot of good work that has been done here that shouldn't go wasted and could instead help kickstart a new platform. Curious if this could possibly be @ intersection of your work responsibilities (even when interpreted broadly)?

Cameron-Micka commented 4 months ago

@Cameron-Micka what are your work responsibilities?

Thank you for the kind works @yosun! I'm currently working on Microsoft Mesh. Which does use Graphics Tools. At the movement Mesh can be run on Microsoft Teams, Windows PC, and Quest. But if/when it runs on VisionOS (specifically spatial apps) that would build a case to start this work.