Remote speech limitations

[sjando]

Mixed Reality OpenXR plugin appears to implement XR_MSFT_holographic_remoting_speech by somehow interfacing with the pre-existing UnityEngine.Windows.Speech APIs. This implementation can cause issues where remote speech doesn't work depending on the speech recognition settings of the machine running the remote application.

To reproduce:

• Open the Basic Sample app from this repo
• Enable 'Holographic Remoting remote app' OpenXR feature for Standalone
• Run the MainMenu scene from Unity, or build and run Standalone
• Use the AppRemoting window to connect to a HL2 running Holographic Remoting Player or similar
• Navigate to the Speech scene
• Observe that speech commands in the Player ('red', 'blue', etc) may or may not work depending on the speech settings of the remote PC

In a non-working scenario we will see in the logs a UnityException: Speech recognition is not supported on this machine. during initialization of speech recognizers. This is not specific to this sample, it is reproducible in any app using Mixed Reality OpenXR Plugin and the issue occurs regardless whether speech recognition is initialized before or after holographic remoting is connected.

This isn't an inherent limitation of remote speech, as remote speech functionality in the native remoting sample app works perfectly fine even in scenarios where Mixed Reality OpenXR plugin fails.

This becomes a real issue when you want to deploy a holographic remoting executable. End-users potentially need to mess around with speech and language settings (potentially requiring Administrator access) on the remote PC to get remote speech working even though this is not a real technical requirement.

I suspect this might be (at least in part) an issue on the Unity side but even if so, a resolution could be to expose the underlying remote speech OpenXR APIs via the Mixed Reality OpenXR plugin in a convenient way. These APIs would be guaranteed to work regardless of speech settings on the remote PC side. This could pair nicely with a 'don't integrate with UnityEngine.Windows.Speech' option on the Holographic Remoting remote app feature which would allow apps to separately make use of speech recognition on both the remote and the player side through the two separate APIs.

[AMollis]

@marlenaklein-msft can you please help investigate this. Note, I've also opened an internal ticket to track this issue.

[AMollis]

@sjando thank you for this issue. Unfortunately, at this time, this issue does not meet our prioritization bar and will not be addressed. If you have any further questions or comments, please provide them via this issue. Thank you.