[BUG] Gray overlay when streaming from device via Miracast or recording video in HoloLens 2

[CZRubik]

Describe the bug

When the app content is being streamed on-line via Miracast or video is recorded with HoloLens 2 there is a gray overlay over real world. Such overlay is not visible in HoloLens 2. Overlay is visible also in development portal (Mixed Reality Capture - MRC).

To reproduce

Steps to reproduce the behavior:

1. Download the MRTKDevTemplate code from https://github.com/MixedRealityToolkit/MixedRealityToolkit-Unity/tree/main/UnityProjects/MRTKDevTemplate
2. Open the project in Unity
3. Set target platform to UWP and architecture ARM64
4. Build the app in Unity
5. Build the code in Visual Studio and deploy the app to the HoloLens 2

Expected behavior

Streamed video doesn't contain the overlay as well the recorded video doesn't contain the overlay.

Your setup

• Unity Version 2021.3.29f1
• MRTK Version 3.0.0 GA

Target platform

• HoloLens 2

Packages in their versions in use

"com.unity.collab-proxy": "2.0.5", "com.unity.ide.rider": "3.0.24", "com.unity.ide.visualstudio": "2.0.18", "com.unity.ide.vscode": "1.2.5", "com.unity.inputsystem": "1.6.3", "com.unity.mobile.android-logcat": "1.3.2", "com.unity.performance.profile-analyzer": "1.2.2", "com.unity.test-framework": "1.1.33", "com.unity.textmeshpro": "3.0.6", "com.unity.timeline": "1.6.5", "com.unity.ugui": "1.0.0", "com.unity.xr.arcore": "5.0.5", "com.unity.xr.arfoundation": "5.0.5", "com.unity.xr.interaction.toolkit": "2.4.0", "com.unity.xr.management": "4.4.0", "com.unity.xr.openxr": "1.8.1", "org.mixedrealitytoolkit.accessibility": "file:../../../org.mixedrealitytoolkit.accessibility", "org.mixedrealitytoolkit.audio": "file:../../../org.mixedrealitytoolkit.audio", "org.mixedrealitytoolkit.core": "file:../../../org.mixedrealitytoolkit.core", "org.mixedrealitytoolkit.data": "file:../../../org.mixedrealitytoolkit.data", "org.mixedrealitytoolkit.diagnostics": "file:../../../org.mixedrealitytoolkit.diagnostics", "org.mixedrealitytoolkit.extendedassets": "file:../../../org.mixedrealitytoolkit.extendedassets", "org.mixedrealitytoolkit.input": "file:../../../org.mixedrealitytoolkit.input", "org.mixedrealitytoolkit.spatialmanipulation": "file:../../../org.mixedrealitytoolkit.spatialmanipulation", "org.mixedrealitytoolkit.standardassets": "file:../../../org.mixedrealitytoolkit.standardassets", "org.mixedrealitytoolkit.tools": "file:../../../org.mixedrealitytoolkit.tools", "org.mixedrealitytoolkit.uxcomponents": "file:../../../org.mixedrealitytoolkit.uxcomponents", "org.mixedrealitytoolkit.uxcomponents.noncanvas": "file:../../../org.mixedrealitytoolkit.uxcomponents.noncanvas", "org.mixedrealitytoolkit.uxcore": "file:../../../org.mixedrealitytoolkit.uxcore", "org.mixedrealitytoolkit.windowsspeech": "file:../../../org.mixedrealitytoolkit.windowsspeech", "com.unity.modules.ai": "1.0.0", "com.unity.modules.androidjni": "1.0.0", "com.unity.modules.animation": "1.0.0", "com.unity.modules.assetbundle": "1.0.0", "com.unity.modules.audio": "1.0.0", "com.unity.modules.cloth": "1.0.0", "com.unity.modules.director": "1.0.0", "com.unity.modules.imageconversion": "1.0.0", "com.unity.modules.imgui": "1.0.0", "com.unity.modules.jsonserialize": "1.0.0", "com.unity.modules.particlesystem": "1.0.0", "com.unity.modules.physics": "1.0.0", "com.unity.modules.physics2d": "1.0.0", "com.unity.modules.screencapture": "1.0.0", "com.unity.modules.terrain": "1.0.0", "com.unity.modules.terrainphysics": "1.0.0", "com.unity.modules.tilemap": "1.0.0", "com.unity.modules.ui": "1.0.0", "com.unity.modules.uielements": "1.0.0", "com.unity.modules.umbra": "1.0.0", "com.unity.modules.unityanalytics": "1.0.0", "com.unity.modules.unitywebrequest": "1.0.0", "com.unity.modules.unitywebrequestassetbundle": "1.0.0", "com.unity.modules.unitywebrequestaudio": "1.0.0", "com.unity.modules.unitywebrequesttexture": "1.0.0", "com.unity.modules.unitywebrequestwww": "1.0.0", "com.unity.modules.vehicles": "1.0.0", "com.unity.modules.video": "1.0.0", "com.unity.modules.vr": "1.0.0", "com.unity.modules.wind": "1.0.0", "com.unity.modules.xr": "1.0.0"

[CZRubik]

Adding some screenshots. 2 with overlay and 1 without.

[AMollis]

@CZRubik this is not an MRTK issue. Please open an issue via the Feedback app on HoloLens. Microsoft will then triage.

However, what you are describing is the expected behavior for applications. I believe the original intent was to simulation the HoloLens opacity level, so that the video viewers' view of the real-world looks similar to the wearer.

[CZRubik]

@AMollis thank you for the feedback I will proceed as proposed. Nevertheless other apps I've made and deployed on HoloLens 2 using e.g. MRTK 2.8 doesn't behave in that sense as you describe and overlay is not there.

[CZRubik]

Hello again, I've reported that issue as recommended using Feedback app.

@AMollis Nevertheless let me accent again that apps build using MRTK 2.8 are not behaving in such sense - means no gray overlay in captured videos/photos nor gray overlay over physical world in MRC.

Therefore I would like to propose to reopen the issue as imho there is lack of justification that the issue is not related to MRTK 3. The different apps (one using MRTK 2.8 and one using MRTK 3) were build on the same computer, tested on the same HoloLens 2 device. Just to avoid some potential doubts, also building an app with the exactly same version of Unity was tested.

[AMollis]

@CZRubik , please answer the following for your MRTK 2.8 app

1. Is it using OpenXR or WMR plugins?
2. If OpenXR, what version of Microsoft Mixed Reality OpenXR plugin are you using?
3. What version of Unity are you using?

[CZRubik]

Hello, thank you for your answer, here the requested information:

1. Is it using OpenXR or WMR plugins? Answer: it uses OpenXR
2. If OpenXR, what version of Microsoft Mixed Reality OpenXR plugin are you using? Answer: 1.7.2
3. What version of Unity are you using? Answer: 2021.3.29f1 same as for the MrtkDevTemplate

[CZRubik]

At this moment no feedback from MS on that matter. Don't know if you have any idea what to do with that. It causes app unusable for our intention to share with audience in the room on the screen, what the user experiences using the HoloLens 2.

[AMollis]

@CZRubik , we looked at this some more. And we found the issue. Sorry for missing this originally.

The camera clear color has an Alpha Value of 255. Changing this to zero should fix your issue.

We're working on fixing this in the default MRTK rig

[CZRubik]

I appreciate your efforts a lot and many thanks for such prompt reaction as always. Will give a try and come back with confirmation within tomorrow.

[CZRubik]

I can confirm, that this has solved the issue and the gray overlay disappeared. Many thanks again for your support.