Immersive application voice commands stop working leaving and resuming

[tletie]

Describe the bug

Immersive application voice commands (utilize UnityEngine.Windows.Speech.KeywordRecognizer) stop working when user open other app like Edge, File Explorer while immersive application still running.

To reproduce

Steps to reproduce the behavior:

• Create any immersive application on Hololens 2 that uses voice commands (utilize UnityEngine.Windows.Speech.KeywordRecognizer).
• Deploy and launch it and keep it running
• Use home gesture and launch Edge or File Explorer
• Come back to the immersive and issue any voice command => won't work. We found other folks having the same issue and mentioned here https://fogbugz.unity3d.com/default.asp?1276443_feij5mjans4g1jds or https://github.com/microsoft/MixedRealityToolkit-Unity/issues/8124 There are suggestions in these articles kind of a walk around is to re-initialized and restart the Speech Recognition objects (KeywordRecognizer, DictationRecognizer, etc.) after the app is restored. Inside OnApplicationFocus(bool focus) and OnApplicationPause(bool pause), the following commands are called UnityEngine.Windows.Speech.PhraseRecognitionSystem.Shutdown(); UnityEngine.Windows.Speech.PhraseRecognitionSystem.Restart(); The issue still there, voice commands is kind of disable when user bring up other app like edge, file explorer while the immersive app still running.

  Expected behavior

While immersive app is running, if we launch other app like edge, file explorer, photo. Then we go back to immersive app and issue voice commands, it should works.

Your setup (please complete the following information)

• Unity Version 2018.4.32f1
• MRTK Version 2.2.0

Target platform (please complete the following information)

• Hololens 2 OS: Version: 21H1 - OS Build: 20348.1018

[ARGs-code]

We appear to be running into this issue. Has this been resolved all?

[rsaru-libre]

@ARGs-code We're able to fix it with the Unity 2019 update but some time updating the HL OS result in the same situation. We tried 2020 LTS and it seems to be working with that.

[ARGs-code]

We are on 2020.3.30f1 and still having this issue.

[rsaru-libre]

@ARGs-code Also I forgot to mention you need to enable and disable the MRTK speech handle.

Here is the code `

private bool registered;

void Awake()
{
    UnityEngine.WSA.Application.windowActivated += ReRegister;
}

private void ReRegister(WindowActivationState state)
{
    switch (state)
    {
        case WindowActivationState.CodeActivated:
            UnRegister();
            Register();
            break;
        case WindowActivationState.Deactivated:
            UnRegister();
            break;
        case WindowActivationState.PointerActivated:
            break;
        default:
            throw new ArgumentOutOfRangeException(nameof(state), state, null);
    }
}

private void UnRegister()
{
    if (registered)
    {
        if (CoreServices.InputSystem != null)
        {
            CoreServices.InputSystem.UnregisterHandler<IMixedRealitySpeechHandler>(this);
        }

        registered = false;
    }
    else
    {
        logger.Log(LogLevel.Error, "Keyword listener already unregistered");
    }
}

private void Register()
{
    if (!registered)
    {
        registered = true;
        CoreServices.InputSystem.RegisterHandler<IMixedRealitySpeechHandler>(this);
    }
    else
    {
        logger.Log(LogLevel.Error, "Keyword listener already registered");
    }
}

Ignore those unrelated logger statement.

[ARGs-code]

Thanks for the suggestion @rsaru-libre

This work around doesn't fix my issue, despite executing all the calls.

[rsaru-libre]

@ARGs-code I've no words how to fix this one then. I fixed this issue using this script and then it works for 2-3 month but after certain hololens os update it stop working and then we looked into the unity2020 and seems to be working in the build. So I assume its something related to the HL OS version which is messing with the voice command.

[ARGs-code]

No problem, it might not be the same issue. It is frustrating that these issues have just been closed, including the one I opened recently. Appreciate the help and your suggestions.

[sos263665629]

@ARGs-code I have used your code, but the problem still exists.

[sos263665629]

I solved the problem using the following script.

`using UnityEngine; using UnityEngine.Windows.Speech;

public class ApplicationBehaviour : MonoBehaviour { private void Awake() { DontDestroyOnLoad(gameObject); }

private void OnApplicationPause(bool pause)
{
    if (!pause)
    {
        //Delayed restart of speech recognition
        //Note: You must delay for a few seconds, otherwise it will not have any effect.
        Invoke("RecongnitionRestart", 5);
    }
}

// restart of speech recognition
private void RecongnitionRestart()
{
    PhraseRecognitionSystem.Shutdown();
    PhraseRecognitionSystem.Restart();
}

}`

[IssueSyncBot]

We appreciate your feedback and thank you for reporting this issue.

Microsoft Mixed Reality Toolkit version 2 (MRTK2) is currently in limited support. This means that Microsoft is only fixing high priority security issues. Unfortunately, this issue does not meet the necessary priority and will be closed. If you strongly feel that this issue deserves more attention, please open a new issue and explain why it is important.

Microsoft recommends that all new HoloLens 2 Unity applications use MRTK3 instead of MRTK2.

Please note that MRTK3 was released in August 2023. It features an all-new architecture for developing rich mixed reality experiences and has a minimum requirement of Unity 2021.3 LTS. For more information about MRTK3, please visit https://www.mixedrealitytoolkit.org.

Thank you for your continued support of the Mixed Reality Toolkit!