Cannot capture this app in video

[dbs2000]

I have deployed the app in my hololens & can run it fine. However, this holograph (the green box) does not get captured in the video that I am trying to take and share with few other folks. I could, however, take video of another full screen app that I created using Unity.. So, it seems that is not about full screen apps in general. Please let me know is there is something that I need to do to have this object get captured in video.

[digiponta]

I need it , too. Or, may the MS Hololens Edge have supported WebGL, before it has been enhanded?

[rachelnicole]

I also need it.

[rachelnicole]

Looking at it a bit more, I'm thinking once you go into record mode, the camera properties get changed and the three.js scene is getting altered based off this recording I was able to capture super quickly.

[digiponta]

Sure, I also can the same video view with my Hololens. The objects go to fly away! However, the shapshot shows the objects except the background.

[Almost-Done]

Sorry all for the late reply. I'l look into this over the next couple of days.

[rachelnicole]

@Almost-Done any luck?

[bryanbocao]

@Almost-Done Thanks for debugging this. Any idea or any thoughts on this issue?

[Almost-Done]

Sorry, no progress yet. A family emergency popped up and I've been away for the last 2 weeks.

[bryanbocao]

No problem, thanks for checking. Any hints on this issue so that we might also help to resolve?

[Almost-Done]

I briefly tried on the emulator and it did not reproduce - capture worked as expected.

This might be caused by something in ANGLE holographic, maybe it interacts with the video capture pipeline. Do you mind trying to switch EGLAutomaticDepthBasedImageStabilizationProperty to false in the sample app's native code? It's in app.cpp line 368

[rachelnicole]

@Almost-Done I'll give it a shot again when I get home tonight. Thanks for taking a look!

[Reda-S]

@Almost-Done it is still not working even after switching EGLAutomaticDepthBasedImageStabilizationProperty to false in the threejs app's native code.

Just to reiterate, if you try to capture just the Holograms, the capture works as expected and you will see the holograms. The problems only happens when you add the PV camera to the capture and then you will not see the holograms anymore.

[rachelnicole]

Yes, live video or pov video don't capture correctly

[Almost-Done]

I'm currently still away with no device available to investigate. Is there an exception thrown on the native side when recording starts? Does the app exit or does it continue with no content visible?

[bryanbocao]

The app did not exit, it continued with content on the HoloLens, but no augmented content on the browser. I accessed the video stream by

https://[HoloLens_IP]/api/holographic/stream/live.mp4

on Chrome This is the relevant log from Visual Studio 2015 when I run my version of ThreeJSApp

'threejsapp.exe' (Win32): Loaded 'U:\Users\DefaultAccount\AppData\Local\DevelopmentFiles\ThreeJSAppVS.Debug_Win32.BryanBo.Cao\threejsapp.exe'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\ntdll.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\KernelBase.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\combase.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\ucrtbase.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\rpcrt4.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\kernel32legacy.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\bcryptprimitives.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'U:\Users\DefaultAccount\AppData\Local\DevelopmentFiles\ThreeJSAppVS.Debug_Win32.BryanBo.Cao\libglesv2.dll'. Cannot find or open the PDB file.
'threejsapp.exe' (Win32): Loaded 'U:\Users\DefaultAccount\AppData\Local\DevelopmentFiles\ThreeJSAppVS.Debug_Win32.BryanBo.Cao\libegl.dll'. Cannot find or open the PDB file.
'threejsapp.exe' (Win32): Loaded 'U:\Programs\WindowsApps\Microsoft.VCLibs.140.00.Debug_14.0.24210.0_x86__8wekyb3d8bbwe\vccorlib140d_app.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'U:\Programs\WindowsApps\Microsoft.VCLibs.140.00.Debug_14.0.24210.0_x86__8wekyb3d8bbwe\vcruntime140d_app.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'U:\Users\DefaultAccount\AppData\Local\DevelopmentFiles\ThreeJSAppVS.Debug_Win32.BryanBo.Cao\ucrtbased.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'U:\Programs\WindowsApps\Microsoft.VCLibs.140.00.Debug_14.0.24210.0_x86__8wekyb3d8bbwe\msvcp140d_app.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\d3d11.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\win32u.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\dxgi.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\D3DCompiler_47.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\oleaut32.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\msvcrt.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\MinUser.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\gdi32.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\gdi32min.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\cryptsp.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\msvcp_win.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\forwarders\kernel32.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\procthreadexthost.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\kernel.appcore.dll'. Symbols loaded.
The thread 0xc38 has exited with code 0 (0x0).
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\twinapi.appcore.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\sechost.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\bcrypt.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\WinTypes.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\SHCore.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\Windows.UI.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\dcomp.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\CoreMessaging.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\actxprxy.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\CoreUIComponents.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\advapi32legacy.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\sspicli.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\MrmCoreR.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\InputHost.dll'. Symbols loaded.
Exception thrown at 0x771A2502 in threejsapp.exe: Microsoft C++ exception: Cn::XH at memory location 0x01CFE81C.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\UiaManager.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\Windows.UI.Core.TextInput.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\TextInputFramework.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\Windows.Mirage.dll'. Cannot find or open the PDB file.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\ntmarta.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\ws2_32.dll'. Symbols loaded.
The thread 0x11c0 has exited with code 0 (0x0).
The thread 0xbf0 has exited with code 0 (0x0).
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\ResourcePolicyClient.dll'. Symbols loaded.
The thread 0x55c has exited with code 0 (0x0).
'threejsapp.exe' (Win32): Unloaded 'C:\Windows\System32\ResourcePolicyClient.dll'
err: egl::GetPlatformDisplayEXT(283): EGL_ANGLE_DISPLAY_ALLOW_RENDER_TO_BACK_BUFFER is deprecated, please useEGL_EXPERIMENTAL_PRESENT_PATH_ANGLE.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\DXGIDebug.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\igd10iumd32.dll'. Cannot find or open the PDB file.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\msvcr120.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\msvcp120.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\ncrypt.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\ntasn1.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\d3d11_3SDKLayers.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\igdusc32.dll'. Cannot find or open the PDB file.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\Windows.Perception.Stub.dll'. Cannot find or open the PDB file.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\cryptbase.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'U:\Users\DefaultAccount\AppData\Local\DevelopmentFiles\ThreeJSAppVS.Debug_Win32.BryanBo.Cao\holojshost.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\Chakra.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\WindowsCodecs.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\mfplat.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\mfreadwrite.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'U:\Programs\WindowsApps\Microsoft.VCLibs.140.00.Debug_14.0.24210.0_x86__8wekyb3d8bbwe\concrt140d_app.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\iertutil.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\cfgmgr32.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\RTWorkQ.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\Windows.Globalization.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\BCP47Langs.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\Windows.ApplicationModel.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\windows.storage.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\powrprof.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\shlwapi.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\profapi.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\propsys.dll'. Symbols loaded.
DXGI WARNING: IDXGIFactory::CreateSwapChain: DXGI_SWAP_CHAIN_DESC.OutputWindow is not a valid window handle. [ MISCELLANEOUS WARNING #65: ]
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\Windows.Web.dll'. Symbols loaded.
log: THREE.WebGLRenderer
Exception thrown at 0x771A2502 in threejsapp.exe: Microsoft C++ exception: Js::RejitException at memory location 0x014FEE2F.
Exception thrown at 0x771A2502 in threejsapp.exe: Microsoft C++ exception: Js::RejitException at memory location 0x019FED2F.
Exception thrown at 0x771A2502 in threejsapp.exe: Microsoft C++ exception: Js::RejitException at memory location 0x019FED2F.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\Windows.Web.Http.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\wininet.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\folderext.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\tokenbinding.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\OnDemandConnRouteHelper.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\IPHLPAPI.DLL'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\winhttp.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\mswsock.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\winnsi.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\nsi.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\FirewallAPI.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\fwbase.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\userenv.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\profext.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\dnsapi.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\urlmon.dll'. Symbols loaded.
'threejsapp.exe' (Win32): Loaded 'C:\Windows\System32\mlang.dll'. Symbols loaded.
log: THREE.Matrix4.getInverse(): can't invert matrix, determinant is 0
log: THREE.Matrix4.getInverse(): can't invert matrix, determinant is 0
Exception thrown at 0x771A2502 in threejsapp.exe: Microsoft C++ exception: Js::RejitException at memory location 0x014FEE2F.
Exception thrown at 0x771A2502 in threejsapp.exe: Microsoft C++ exception: Js::RejitException at memory location 0x014FEE2F.
Exception thrown at 0x771A2502 in threejsapp.exe: Microsoft C++ exception: Js::RejitException at memory location 0x019FED2F.
Exception thrown at 0x771A2502 in threejsapp.exe: Microsoft C++ exception: Js::RejitException at memory location 0x014FEE2F.
Exception thrown at 0x771A2502 in threejsapp.exe: Microsoft C++ exception: Js::RejitException at memory location 0x019FED2F.
The thread 0x16b4 has exited with code 0 (0x0).
The thread 0x1010 has exited with code 0 (0x0).
The thread 0xd4c has exited with code 0 (0x0).
The thread 0x11e4 has exited with code 0 (0x0).
'threejsapp.exe' (Win32): Unloaded 'C:\Windows\System32\mlang.dll'
Exception thrown at 0x771A2502 (KernelBase.dll) in threejsapp.exe: 0x40080201: WinRT originate error (parameters: 0x80072F78, 0x00000039, 0x018FF0D8).
Exception thrown at 0x771A2502 in threejsapp.exe: Microsoft C++ exception: Platform::COMException ^ at memory location 0x01CFE108. HRESULT:0x80072F78 The text associated with this error code could not be found.
WinRT information: The server returned an invalid or unrecognized response
Stack trace:
 >[External Code]
The program '[324] threejsapp.exe' has exited with code 0 (0x0).

I terminated the app from the Visual Studio 2015 in the end.

[Almost-Done]

@BryanBo-Cao Can you try recording instead of live streaming? You can start it from the device portal, on the Mixed Reality Capture page.

[rachelnicole]

@almost-done recording is the same thing, there's an example of what happens in the gif above

[bryanbocao]

@Almost-Done Thanks for checking this issue! I tried setting EGLAutomaticDepthBasedImageStabilizationProperty false in app.cpp in Threejsapp

    if (mStationaryReferenceFrame != nullptr)
    {
        surfaceCreationProperties->Insert(ref new String(EGLBaseCoordinateSystemProperty), mStationaryReferenceFrame);
        surfaceCreationProperties->Insert(ref new String(EGLAutomaticStereoRenderingProperty), PropertyValue::CreateBoolean(true));
        //surfaceCreationProperties->Insert(ref new String(EGLAutomaticDepthBasedImageStabilizationProperty), PropertyValue::CreateBoolean(true));
        surfaceCreationProperties->Insert(ref new String(EGLAutomaticDepthBasedImageStabilizationProperty), PropertyValue::CreateBoolean(false));
    }

but got the same result, no contents in neither livestreaming or recording in the browser.

[digiponta]

Can I also understand the issue by studying the pages?: https://developer.microsoft.com/en-us/windows/mixed-reality/locatable_camera#device_camera_information

[bryanbocao]

On a real HoloLens (not emulator), there were a couple of phenomena I've seen with my ThreeJSApp, when it rendered the objects very slowly(I could feel the objects' movements and lights changing frame by frame in HoloLens), I happened to see the rendered objects in a browser from video streaming that they slowly faded out to the bottom-right direction. Like the gif shown by @rachelnicole above in a slow version.

[Reda-S]

was anyone able to solve the video capture issue?

[dagstjerna]

Hi, I got the same issue running my own webGL-engine inside HoloJS. As soon as I open the HoloJS app the recording loses the holographic stream and only camera stream continues to work. I have a presentation next week and really needs a solution for this.

I also use the hardware and not the emulator. Best Regards

[Almost-Done]

I raised this issue with folks on the HoloLens team. Hopefully there will be progress in the next couple of days.

Meanwhile, can someone grab a log while reproducing the issue. Instructions:

1. On device portal, go to the Logging page
2. Enable the Microsoft-Windows-Mirage-MixedRealityCapture provider from the list of registered providers
3. Reproduce the issue.
4. On the Logging page, click on "Save to file" and share the resulting log file

[Reda-S]

Hi Cristi,

I followed your instructions:

1-Enabled Microsoft-Windows-Mirage-MixedRealityCapture provider (5-Detailed Trace) 2- Started mixed reality capture "Live preview" Holograms + PVCamera 3- Run a holojs based app (no holograms showed on the live preview ) 4-stopped the live preview and saved the trace file

EtwTrace.txt

[Almost-Done]

Thank you Reda for the trace. And my apologies to everybody for the delay.

I pushed a workaround to develop/fix-holograms-capture. Please let me know if it fixes the issue.

I believe I understand the root cause: HoloJS has a feature called "depth based LSR" that if enabled automatically sets the focus point to stabilize the objects in the scene. Mixed reality capture also uses this focus point; if the focus point is set behind the camera, the holograms disappear from the video capture (but are still rendered on the screen).

The workaround is disabling depth based LSR. Previously I suggested disabling the depth based LSR and it did not fix the issue. However, there is a bug in ANGLE-Holographic and it does not respect the disabling of depth based LSR. I checked in fixed ANGLE binaries and disabled depth based LSR in the ThreeJS sample app.

If the workaround provided works, we can fix the real bug of setting the focus point incorrectly in depth based LSR.

[Reda-S]

Thank you Cristi, the provided workaround is working!

[Almost-Done]

Great.

I'll close this issue and open another one for "depth based LSR not setting the focus point correctly".