"Select" voice command does not fire the appropriate events when using OpenXR on HoloLens 2

[cj-lopez]

Describe the bug

When you Gaze and say "Select" on a HoloLens 2, the MRTK input system should fire the appropriate events for the "select" action. This no longer works like it did for the Legacy XR. Other keywords work just fine.

To reproduce

Steps to reproduce the behavior:

The easiest way to see the problem is with the EyeTrackingDemo example

1. Install the OpenXR Plugin
2. Import the EyeTracking demo
3. Deploy to the HoloLens 2
4. In the TargetSelection scene, say "select" while looking at a gem, the speech confirmation tooltip appears, but the gem does not explode. Saying "explode" does explode the gem.
5. Gaze at a different scene in the Scene Menu and say "select". It's no longer possible to change scenes with the "select command".

Expected behavior

Saying "Select" should fire the appropriate events to cause a "select" action.

Screenshots

Unfortunately, the "select" tooltip isn't captured when I try to take a screenshot.

Your setup (please complete the following information)

• Unity Version 2020.3.15f2
• MRTK Version 2.7.2
• MR OpenXR Plugin 1.0.2
• OpenXR Plugin 1.2.8
• XR Plugin Management 4.0.7

Target platform (please complete the following information)

• HoloLens 2

[elbuhofantasma]

Hello @cj-lopez Thanks for reporting this speech issue! I was able to reproduce this across Unity versions, MRTK versions, and across XR pipelines. @cre8ivepark will investigate this in the near future and we will give you an update when the investigation is complete.

[cj-lopez]

@elbuhofantasma @cre8ivepark Is this also related - [https://github.com/microsoft/MixedRealityToolkit-Unity/issues/8582]?

[elbuhofantasma]

@cj-lopez I do not believe that issue #8582 is related to this issue. The "Select" voice command in this issue seems to be occurring due the MRTK not being detected by the EyeTrackingTarget.cs script as seen in this picture below in the Eye Tracking Targeting example scene. This issue occurs in the OpenXR and XR SDK pipelines and across Unity versions. I have not tried to reproduce this issue on Unity 2018 LTS yet.

[Spoon2311]

Any updates on this issue?

[Spoon2311]

Any updates on this issue?

[cre8ivepark]

Verified that this is not only for the Eye Tracking example scenes. I don't see a gaze cursor even if when 'Select' is recognized. It does not trigger any target objects (e.g. buttons in Hand Interaction Example scene)

[ghost]

Is there any update on this issue?

[Spoon2311]

Hey guys @keveleigh are there any updates available to this issue?

[abhimanyusinghal]

Folks @keveleigh Is there any update? This is causing serious issues..

[keveleigh]

@DavidShapiraRimon @Spoon2311 @abhimanyusinghal

No concrete updates at the moment, but it is something I'm starting to investigate supporting in the Mixed Reality OpenXR Plugin. The one restriction will likely be that it doesn't work over remoting (but I'm also working on better support for the keyword recognizer over OpenXR remoting, which should balance out in many cases). Would that be a dealbreaker?

[Spoon2311]

Hey @keveleigh For myself the most important part is, that the eye tracking will work again when deploying the application to hololens.

If remoting won't support it it is totally fine for me.

[MaxWang-MS]

The fix for this issue has been released in MRTK 2.8.1. Please let me know if you are still running into the issue!

[bnelligan]

The fix in PR 10661 does not work for MRTK 2.8.2. After much debugging, I found a fix for our project with the following changes:

• Add the EyeGazeInteractionProfile to the OpenXR Interaction Profiles setting. (Source: Joost van Schaik)
• Update line 134 of OpenXRDeviceManager.cs as follows: Service?.RaiseSpeechCommandRecognized(controller.InputSource, RecognitionConfidenceLevel.High, TimeSpan.MinValue, DateTime.Now, new SpeechCommands("Select", KeyCode.Alpha1, new MixedRealityInputAction(1, "Select", AxisType.Digital)));

[quater782]

The fix in PR 10661 does not work for MRTK 2.8.2. After much debugging, I found a fix for our project with the following changes:

• Add the EyeGazeInteractionProfile to the OpenXR Interaction Profiles setting. (Source: Joost van Schaik)
• Update line 134 of OpenXRDeviceManager.cs as follows: Service?.RaiseSpeechCommandRecognized(controller.InputSource, RecognitionConfidenceLevel.High, TimeSpan.MinValue, DateTime.Now, new SpeechCommands("Select", KeyCode.Alpha1, new MixedRealityInputAction(1, "Select", AxisType.Digital)));

This works perfectly for me, thanks a lot for the help.