The GrammarRecognizer of UnityEngine.Windows.Speech seems not to work within Unity 2020.3.6f1 and MRTK 2.6.1.

powerschaf commented 3 years ago

Describe the bug

To reproduce

Steps to reproduce the behavior:

Make Unity Project / Switch to UWP
Add MRTK 2.6.1 , add Windows Speech Input Provider
Use XRSDK profil for HoloLens2 , Install XR Plugin
Make sure Microphone is enabled in Player Settings
TestClass: `using UnityEngine.Windows.Speech;

public class test : MonoBehaviour { private GrammarRecognizer m_recognizer; public GrammarRecognizer Recognizer { get => m_recognizer; set => m_recognizer = value; }

void Start()
{
    Recognizer = new GrammarRecognizer(Application.streamingAssetsPath + "/SRGS/test.xml");
    Recognizer.OnPhraseRecognized += Recognizer_OnPhraseRecognized;
    Recognizer.Start();
}

private void Recognizer_OnPhraseRecognized(PhraseRecognizedEventArgs args)
{
    Debug.Log("I recognized something");
}

void Update()
{
    if(Recognizer.IsRunning)
    {
        Debug.Log("Running");
    }
}

}`

Make simple SRGS XML File:

Name it test.xml and put it in the Assets/StreamingAssets/SRGS Folder
Hit Play and say "Player" .... Nothing happens (No errors appear, the file is loaded correctly)

Expected behavior

The word player is recognized and in the Unity Editor Debug.Log("I recognized something"); is fired

Your setup (please complete the following information)

Unity 2020.3.6f1 and MRTK 2.6.1.

Target platform (please complete the following information)

HoloLens 2

Additional context

Is it possible to register a huge amount of strings to the KeywordRecognizer? In my app the user says for example "bloodpressure 120 to 80" and other medical stuff, i need to get the numbers into TextLabels... If i have to register everything as one Keyword with all numbers that might be too much?? I don't know, if its no problem i will do it like that.

jasonhbartlett commented 3 years ago

Hello, I'm also having a problem with UnityEngine.Windows.Speech namespace working with Windows Mixed Reality/OpenXR.

I'm actually not even using MRTK. Also my target platform is Standalone>Windows. I'm building a VR training app with Unity 2020.3 and OpenXR. I've added the Mixed Reality OpenXR plugins with the MS Mixed Reality Feature Tool.

In my app the UnityEngine.Windows.Speech KeywordRecognizer works perfectly EXCEPT when I have a WMR headset connected and WMR Portal running.

So it works in the editor, as a desktop app (without WMR headset), and it works running on Oculus Quest (via Link).

But if I use a WMR headset (test using both Reverb G1 and G2), it's as if the public event PhraseRecognizedDelegate OnPhraseRecognized from UnityEngine.Windows.Speech doesn't exist.

So I'm not sure if this is an issue with that namespace which is part of the UnityEngine.CoreModule.dll or has something to do with the Mixed Reality OpenXR plugin. Or something else.

I've posted in the Unity Forum under the topic of VR as a potential bug. My WMR VR OpenXR Desktop Unity App Speech Commands Don't Work on Headset

@MaxWang-MS Any thoughts if this is indeed a bug with thew WMR, with Mixed Reality OpenXR plugins, or with the UnityEngine.CoreModule.dll? If it's a Unity bug then it should be submitted to the Unity Issue Tracker. If it's something with WMR or OpenXR then I don't know where to report it.

MaxWang-MS commented 2 years ago

Hi @powerschaf, sorry for the delayed response. The issue you described seems to be a Unity bug. Could you file a bug with Unity by going to Help (in the Unity editor menu bar) → Report a bug... and post a link to the Unity Issue Tracker page once it's created so that the community can get updates on the bug? Thanks!

Regarding your need to register a large amount of strings for the use case you mentioned, besides using the grammar recognizer provided by Unity you could also leverage the Azure Speech SDK, which has support for Unity and pattern matching.

MaxWang-MS commented 2 years ago

Hi @jasonhbartlett, looking at the link you posted the issue seems to be tracked by both the Unity forum post and a OpenXR sample repo issue, so please rest assured the issue you are seeing is being handled by the right people.

stale[bot] commented 2 years ago

This issue has been marked as stale by an automated process because it has not had any recent activity. It will be automatically closed in 30 days if no further activity occurs. If this is still an issue please add a new comment with more recent details and repro steps.

IssueSyncBot commented 9 months ago

We appreciate your feedback and thank you for reporting this issue.

Microsoft Mixed Reality Toolkit version 2 (MRTK2) is currently in limited support. This means that Microsoft is only fixing high priority security issues. Unfortunately, this issue does not meet the necessary priority and will be closed. If you strongly feel that this issue deserves more attention, please open a new issue and explain why it is important.

Microsoft recommends that all new HoloLens 2 Unity applications use MRTK3 instead of MRTK2.

Please note that MRTK3 was released in August 2023. It features an all-new architecture for developing rich mixed reality experiences and has a minimum requirement of Unity 2021.3 LTS. For more information about MRTK3, please visit https://www.mixedrealitytoolkit.org.

Thank you for your continued support of the Mixed Reality Toolkit!

microsoft / MixedRealityToolkit-Unity