aleneum commented 4 years ago

I try to use SpectatorView together with WebRTC. Together as in "as part of the same Unity app", not at the same time. It seems that when one service has initialized the MediaCapture, the other one cannot do it afterwards unless I restart the app. So even after ArUcoMarkerDetector has been released and HoloLensCamera has been disposed, WebRTC fails to get a hold of the capture. Similar issue when I first start WebRTC, destroy the GameObject and then try to ArUcoMarkerDetector. Both services work fine in conjunction with Vuforia. I can switch back and forth without any issues.

System

Unity: 2019.2.19f1
WebRTC: https://github.com/microsoft/MixedReality-WebRTC/commit/73b69d7da6c74cb02b97e5ec1a627e7ce07eba55
SpectatorView: 8f7d76c9de6542a5fb4c0b4889efbb4916bfb481
HoloLens: 17763.973.x86fre.rs5_release_svc_prod1.200103-1310

Scenario 1: Spectator View first, WebRTC second

Initializing `HoloLensCamera`

StateSynchronizationBroadcaster: Broadcaster received connection from xxx.xxx.xxx.xxx.
SpatialCoordinateSystemManager: Creating new SpatialCoordinateSystemParticipant, IPAddress: xxx.xxx.xxx.xxx, DebugLogging: True
SpatialCoordinateSystemManager: Invoking ParticipantConnected event
SpatialCoordinateWorldOriginTransformer: Participant connected: xxx.xxx.xxx.xxx
SpatialCoordinateSystemManager: LocalizeMessageReceived
CameraStateManager: Entering State 'SpectatorView'
MarkerVisualDetectorSpatialLocalizer.LocalizationSession: Waiting for marker visual, CanBeCanceled:True, IsCancellationRequested:False
SpatialCoordinateSystemManager: Data received for participant: xxx.xxx.xxx.xxx, LocalizationDataExchange
MarkerVisualDetectorSpatialLocalizer.LocalizationSession: Received command: MVISUALDISC
MarkerVisualDetectorSpatialLocalizer.LocalizationSession: GetMarkerId currently returns 1 when possible to avoid conficts with the MarkerDetectorSpatialLocalizer that uses 0. Additional work is still required to enable assigning unique marker ids to different application participants.
MarkerVisualDetectorSpatialLocalizer.LocalizationSession: Sending coordinate assignment: 1
MarkerVisualDetectorSpatialLocalizer.LocalizationSession: Attempting to discover coordinate: 1, CanBeCanceled:True, IsCancellationRequested:False
MarkerDetector: Starting detection
ArUcoMarkerDetector: Starting ArUco marker detection
ArUcoMarkerDetector: Setting up HoloLensCamera
MarkerDetector: Marker with id: 1 wasn't found yet.
MarkerDetector: Waiting another second
ArUcoMarkerDetector: HoloLensCamera initialized
Exception thrown at 0x76EA3072 (KernelBase.dll) in Avikom.exe: WinRT originate error - 0xC00DABE0 : 'No capture devices are available.'.
ArUcoMarkerDetector: HoloLensCamera successfully started
ArUcoMarkerDetector: Image obtained from HoloLens

Releasing `HoloLensCamera`

MarkerDetector: Markers Updated
MarkerDetector: Processed all markers
MarkerDetector: Marker 1 not seen this time.
MarkerDetector: Found all markers
ArUcoMarkerDetector: Stopping ArUco marker detection
ArUcoMarkerDetector: Cleaning up HoloLensCamera
Exception thrown at 0x76EA3072 (KernelBase.dll) in Avikom.exe: WinRT originate error - 0x80000013 : 'The object has been closed.'.
Exception thrown at 0x76EA3072 in Avikom.exe: Microsoft C++ exception: Il2CppExceptionWrapper at memory location 0x019FF2A0.
Exception thrown at 0x76EA3072 (KernelBase.dll) in Avikom.exe: WinRT originate error - 0x80000013 : 'System.ObjectDisposedException: The object has been closed.

The object has been closed.

  at Windows.Media.Devices.VideoDeviceController.get_IsoSpeedControl () [0x00000] in <00000000000000000000000000000000>:0 
  at Microsoft.MixedReality.PhotoCapture.HoloLensCamera.get_Gain () [0x00000] in <00000000000000000000000000000000>:0 
  at Microsoft.MixedReality.PhotoCapture.HoloLensCamera.GetFrameFromMediaFrameReader (Windows.Media.Capture.Frames.MediaFrameReader frameReader) [0x00000] in <000000000000000'.
Exception thrown at 0x76EA3072 (KernelBase.dll) in Avikom.exe: 0x40080202: WinRT transform error (parameters: 0x80000013, 0x00000000, 0x00000027, 0x019FEF60).
MarkerDetector: Stopped detection
MarkerVisualDetectorSpatialLocalizer.LocalizationSession: Coordinate discovery completed: 1
MarkerVisualDetectorSpatialLocalizer.LocalizationSession: Sending coordinate found: 1
MarkerDetector: Disposed
CameraStateManager: Entering State 'Unallocated'

Initializing `MediaCapture` in `PeerConnection`

CameraStateManager: Entering State 'WebRTC'
Exception thrown at 0x76EA3072 (KernelBase.dll) in Avikom.exe: WinRT originate error - 0xC00D36BB : 'An unexpected error has occurred in the operation requested.'.
Exception thrown at 0x76EA3072 in Avikom.exe: Microsoft C++ exception: Il2CppExceptionWrapper at memory location 0x01CFED14.
Exception: An unexpected error has occurred in the operation requested.
An unexpected error has occurred in the operation requested.
  at Windows.Media.Capture.MediaCapture.InitializeAsync (Windows.Media.Capture.MediaCaptureInitializationSettings mediaCaptureInitializationSettings) [0x00000] in <00000000000000000000000000000000>:0 
  at Microsoft.MixedReality.WebRTC.Unity.PeerConnection.RequestAccessAndInitAsync (System.Threading.CancellationToken token) [0x00000] in <00000000000000000000000000000000>:0 
  at Microsoft.MixedReality.WebRTC.Unity.PeerConnection+<>c__DisplayClass16_0.<InitializeAsync>b__0 () [0x00000] in <00000000000000000000000000000000>:0 
  at UnityEngine.WSA.AppCallbackItem.Invoke () [0x00000] in <00000000000000000000000000000000>:0 

(Filename: currently not available on il2cpp Line: -1)

Cross thread invocation callback threw an exception. Exception: An unexpected error has occurred in the operation requested.
An unexpected error has occurred in the operation requested.
  at Windows.Media.Capture.MediaCapture.InitializeAsync (Windows.Media.Capture.MediaCaptureInitializationSettings mediaCaptureInitializationSettings) [0x00000] in <00000000000000000000000000000000>:0 
  at Microsoft.MixedReality.WebRTC.Unity.PeerConnection.RequestAccessAndInitAsync (System.Threading.CancellationToken token) [0x00000] in <00000000000000000000000000000000>:0 
  at Microsoft.MixedReality.WebRTC.Unity.PeerConnection+<>c__DisplayClass16_0.<InitializeAsync>b__0 () [0x00000] in <00000000000000000000000000000000>:0 
  at UnityEngine.WSA.AppCallbackItem.Invoke () [0x00000] in <00000000000000000000000000000000>:0 

(Filename: currently not available on il2cpp Line: -1)

Found video capture device 'MN34150' (id:\\?\DISPLAY#INT22B8#4&27b432bd&0&UID139960#{e5323777-f976-4f5b-9b55-b94699c46e44}\{CDD6871A-56CA-4386-BAE7-D24B564378A9}).
CameraStateManager: Entering State 'Unallocated'

Scenario 2: WebRTC first. SpectatorView second

Initializing `MediaCapture` in `PeerConnection`

CameraStateManager: Entering State 'WebRTC'
Initializing WebRTC plugin...
WebRTC plugin initialized successfully.
Found video capture device 'MN34150' (id:\\?\DISPLAY#INT22B8#4&27b432bd&0&UID139960#{e5323777-f976-4f5b-9b55-b94699c46e44}\{CDD6871A-56CA-4386-BAE7-D24B564378A9}).
VideoEffect Added
Exception thrown at 0x76EA3072 (KernelBase.dll) in Avikom.exe: WinRT originate error - 0xC00D36B3 : 'The stream number provided was invalid.'.

Destroying WebRTC GameObject (or disabling it)

Exception thrown at 0x76EA3072 (KernelBase.dll) in Avikom.exe: WinRT originate error - 0xC00D36B5 : 'The callee is currently not accepting further input.'.
VideoEffect removed
CameraStateManager: Entering State 'Unallocated'

Initializing `HoloLensCamera`

MarkerDetector: Starting detection
ArUcoMarkerDetector: Starting ArUco marker detection
ArUcoMarkerDetector: Setting up HoloLensCamera
MarkerDetector: Marker with id: 1 wasn't found yet.
MarkerDetector: Waiting another second
ArUcoMarkerDetector: HoloLensCamera initialized
Exception thrown at 0x76EA3072 (KernelBase.dll) in Avikom.exe: 0x8001010E: The application called an interface that was marshalled for a different thread.
Exception thrown at 0x76EA3072 in Avikom.exe: Microsoft C++ exception: Il2CppExceptionWrapper at memory location 0x026DDD94.
Exception thrown at 0x76EA3072 in Avikom.exe: Microsoft C++ exception: Il2CppExceptionWrapper at memory location 0x026DEEE0.
Exception: Exception of type 'System.Exception' was thrown.
  at Windows.Media.Capture.MediaCapture..ctor () [0x00000] in <00000000000000000000000000000000>:0 
  at Microsoft.MixedReality.PhotoCapture.StreamSelector.set_StreamDescriptions (System.Collections.Generic.List`1[T] value) [0x00000] in <00000000000000000000000000000000>:0 
  at Vuforia.VuforiaConfiguration+WebCamConfiguration.get_TurnOffWebCam () [0x00000] in <00000000000000000000000000000000>:0 
  at Microsoft.MixedReality.PhotoCapture.HoloLensCamera.Start (Microsoft.MixedReality.PhotoCapture.StreamDescription streamDesc) [0x00000] in <00000000000000000000000000000000>:0 
  at Microsoft.MixedReality.SpectatorView.ArUcoMarkerDetector.CameraInitialized (Microsoft.MixedReality.PhotoCapture.HoloLensCamera sender, System.Boolean initializeSuccessful) [0x00000] in <00000000000000000000000000000000>:0 
  at Microsoft.MixedReality.PhotoCapture.OnCameraInitializedHandler.Invoke (Microsoft.MixedReality.PhotoCapture.HoloLensCamera sender, System.Boolean initializeSuccessful) [0x00000] in <00000000000000000000000000000000>:0 
  at Microsoft.MixedReality.PhotoCapture.StreamSelector.set_StreamDescriptions (System.Collections.Generic.List`1[T] value) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.InvokeMoveNext (System.Object stateMachine) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Threading.ContextCallback.Invoke (System.Object state) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () [0x00000] in <00000000000000000000000000000000>:0 
  at System.Action.Invoke () [0x00000] in <00000000000000000000000000000000>:0 
  at System.Threading.Tasks.SynchronizationContextAwaitTaskContinuation+<>c.<.cctor>b__7_0 (System.Object state) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Threading.SendOrPostCallback.Invoke (System.Object state) [0x00000] in <00000000000000000000000000000000>:0 
  at UnityEngine.WSA.WindowSizeChanged..ctor (System.Object object, System.IntPtr method) [0x00000] in <00000000000000000000000000000000>:0 
  at UnityEngine.UnitySynchronizationContext.Exec () [0x00000] in <00000000000000000000000000000000>:0 
  at UnityEngine.UnitySynchronizationContext.ExecuteTasks () [0x00000] in <00000000000000000000000000000000>:0 
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x00000] in <00000000000000000000000000000000>:0 
  at System.Runtime.CompilerServices.AsyncMethodBuilderCore+<>c.<ThrowAsync>b__6_0 (System.Object state) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Threading.SendOrPostCallback.Invoke (System.Object state) [0x00000] in <00000000000000000000000000000000>:0 
  at UnityEngine.WSA.WindowSizeChanged..ctor (System.Object object, System.IntPtr method) [0x00000] in <00000000000000000000000000000000>:0 
  at UnityEngine.UnitySynchronizationContext.Exec () [0x00000] in <00000000000000000000000000000000>:0 
  at UnityEngine.UnitySynchronizationContext.ExecuteTasks () [0x00000] in <00000000000000000000000000000000>:0 
UnityEngine.DebugLogHandler:Internal_LogException(Exception, Object)
UnityEngine.DebugLogHandler:LogException(Exception, Object)
UnityEngine.Logger:LogException(Exception, Object)
UnityEngine.Debug:LogException(Exception)
UnityEngine.WSA.WindowSizeChanged:.ctor(Object, IntPtr)
UnityEngine.UnitySynchronizationContext:Exec()
UnityEngine.UnitySynchronizationContext:ExecuteTasks()

(Filename: currently not available on il2cpp Line: -1)

MarkerDetector: Marker with id: 1 wasn't found yet.
MarkerDetector: Waiting another second

aleneum commented 4 years ago

First thing I will do is rebuilding both services with the most recent version. Second step will be to create a Unity project without Vuforia to make sure that the problem is actually related to MediaCapture in SpectatorView and WebRTC and not also Vuforia.

aleneum commented 4 years ago

Tested it with

Unity 2019.2.19f
https://github.com/microsoft/MixedReality-WebRTC/commit/0ad9423ac856d1426cb043404ecedf93e85406c6
160dd4acea9d4d393941db486c3177f2983293e9

Procedure

Cloned and built WebRTC
Added SpectatorView to demo Unity project (libs\Microsoft.MixedReality.WebRTC.Unity) as instructed in the Readme.
Built 3rd-party OpenCV and dependencies
Compiled native libraries and let the script copy the results to libs\Microsoft.MixedReality.WebRTC.Unity/Assets
Successfully tested SpectatorView with Hololens and Android
In VideoChatDemo:
- Moved VideoChatDemo and SceneVideoPlayer to new empty object WebRTCGroup
- Created SpectatorViewGroup to Scene and added SceneViewer prefab
- Created Script which destroys WebRTCGroup 5 seconds after start (this is enough time for WebRTC to initialize the local camera) and attached it to SpectatorViewGroup
Built project and ran solution on HoloLens
Ran MixedRealityWebRTCUnityDemo on Android to connect to PeerConnection and trigger MediaCapture.InitializeAsync

Result

Same as described above:

Setting up 1 worker threads for Enlighten.
  Thread -> id: 428 -> priority: 1 
A Canvas named "NodeDssSignalerControls" with a render mode of ScreenSpaceOverlay is being used while VR is also enabled. This Canvas will continue to incur a rendering cost, but will not be visible while in VR. Is this intentional?

(Filename: C:\buildslave\unity\build\Modules/UI/CanvasManager.cpp Line: 948)

VideoEffect Added
Exception thrown at 0x76EA3072 (KernelBase.dll) in MixedRealityWebRTCUnityDemo.exe: WinRT originate error - 0xC00D36B3 : 'The stream number provided was invalid.'.
Curl error 7: Failed to connect to 127.0.0.1 port 3000: Connection refused

(Filename: C:\buildslave\unity\build\Modules/UnityWebRequest/Implementations/TransportCurl.cpp Line: 799)

Network error trying to send data to http://127.0.0.1:3000/: Cannot connect to destination host
UnityEngine.DebugLogHandler:Internal_Log(LogType, LogOption, String, Object)
UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
UnityEngine.Logger:Log(LogType, Object)
UnityEngine.Debug:LogError(Object)
Microsoft.MixedReality.WebRTC.Unity.<CO_GetAndProcessFromServer>d__15:MoveNext()
UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)

(Filename: C:\buildslave\unity\build\Runtime/Export/Debug/Debug.bindings.h Line: 35)

[...]

Exception thrown at 0x76EA3072 (KernelBase.dll) in MixedRealityWebRTCUnityDemo.exe: WinRT originate error - 0xC00D36B5 : 'The callee is currently not accepting further input.'.
VideoEffect removed
Curl error 7: Failed to connect to 127.0.0.1 port 3000: Connection refused

(Filename: C:\buildslave\unity\build\Modules/UnityWebRequest/Implementations/TransportCurl.cpp Line: 799)

Server connected to xxx.xxx.xxx.xxx
UnityEngine.DebugLogHandler:Internal_Log(LogType, LogOption, String, Object)
UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
UnityEngine.Logger:Log(LogType, Object)
UnityEngine.Debug:Log(Object)
Microsoft.MixedReality.SpectatorView.TCPConnectionManager:OnServerConnected(SocketerClient, Int32, String)
System.Action`3:Invoke(T1, T2, T3)
Microsoft.MixedReality.SpectatorView.SocketerClient:TcpServer_Connect(Object, ConnectionEventArgs)
System.EventHandler`1:Invoke(Object, TEventArgs)
Microsoft.MixedReality.SpectatorView.SocketServer:OnConnect(Socket)
Microsoft.MixedReality.SpectatorView.SocketServer:ClientThreadFunc(Object)
System.Threading.ParameterizedThreadStart:Invoke(Object)
System.Threading.ThreadHelper:ThreadStart_Context(Object)
System.Threading.ContextCallback:Invoke(Object)
System.Threading.ExecutionContext:RunInternal(ExecutionContext, ContextCallback, Object, Boolean)
System.Threading.ExecutionContext:Run(ExecutionContext, ContextCallback, Object, Boolean)
System.Threading.ExecutionContext:Run(ExecutionContext, ContextCallback, Object)
System.Threading.ThreadHelper:ThreadStart(Object)
System.Threading.ParameterizedThreadStart:Invoke(Object)

(Filename: C:\buildslave\unity\build\Runtime/Export/Debug/Debug.bindings.h Line: 35)

[...]

MarkerDetector: Starting detection
UnityEngine.DebugLogHandler:Internal_Log(LogType, LogOption, String, Object)
UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
UnityEngine.Logger:Log(LogType, Object)
UnityEngine.Debug:Log(Object)
Microsoft.MixedReality.SpectatorView.MarkerDetectorCoordinateService:DebugLog(String)
Microsoft.MixedReality.SpectatorView.<EnumerateAllAssets>d__0:.ctor(Int32)
System.Array:UnsafeStore(T[], Int32, T)
Microsoft.MixedReality.SpectatorView.MarkerDetectorCoordinateService:OnDiscoverCoordinatesImplAsync(CancellationToken, Int32[])
Microsoft.MixedReality.SpectatorView.<EnumerateAllAssets>d__0:.ctor(Int32)
System.Array:UnsafeStore(T[], Int32, T)
Microsoft.MixedReality.SpectatorView.MarkerDetectorCoordinateService:OnDiscoverCoordinatesAsync(CancellationToken, Int32[])
Microsoft.Azure.SpatialAnchors.<System-Collections-IEnumerable-GetEnumerator>d__26:.ctor(Int32)
System.Array:UnsafeStore(T[], Int32, T)
Microsoft.MixedReality.SpatialAlignment.SpatialCoordinateServiceBase`1:TryDiscoverCoordinatesAsync(CancellationToken, TKey[])
Microsoft.MixedReality.SpatialAlignment.SpatialCoordinateServiceBase`1:Microsoft.MixedReality.SpatialAlignment.ISpatialCoordinateService.TryDiscoverCoordinatesAsync(CancellationToken, String[])
Microsoft.MixedReality.SpectatorView.<EnumerateAllAssets>d__0:.ctor(Int32)
System.Runtime.CompilerServices.MoveNextRunner:InvokeMoveNext(Object)
System.Threading.ContextCallback:Invoke(Object)
System.Threading.ExecutionContext:RunInternal(ExecutionContext, ContextCallback, Object, Boolean)
System.Threading.ExecutionContext:Run(ExecutionContext, ContextCallback, Object, Boolean)
System.Runtime.CompilerServices.MoveNextRunner:Run()
System.Action:Invoke()
System.Threading.Tasks.AwaitTaskContinuation:InvokeAction(Object)
System.Threading.ContextCallback:Invoke(Object)
System.Threading.Tasks.AwaitTaskContinuation:RunCallback(ContextCallback, Object, Task&)
System.Threading.Tasks.SynchronizationContextAwaitTaskContinuation:Run(Task, Boolean)
System.Threading.Tasks.Task:FinishContinuations()
System.Threading.Tasks.Task:FinishStageThree()
System.Threading.Tasks.Task`1:TrySetResult(TResult)
System.Threading.Tasks.CompleteOnInvokePromise:Invoke(Task)
System.Threading.Tasks.Task:FinishContinuations()
System.Threading.Tasks.Task:FinishStageThree()
System.Threading.Tasks.Task`1:TrySetResult(TResult)
System.Threading.Tasks.TaskCompletionSource`1:TrySetResult(TResult)
Microsoft.MixedReality.SpectatorView.LocalizationSession:OnDataReceived(BinaryReader)
Microsoft.MixedReality.SpectatorView.SpatialCoordinateSystemManager:OnParticipantDataReceived(INetworkConnection, String, BinaryReader, Int32)
Microsoft.MixedReality.SpectatorView.CommandHandler:Invoke(INetworkConnection, String, BinaryReader, Int32)
Microsoft.MixedReality.SpectatorView.CommandRegistry`1:NotifyCommand(INetworkConnection, String, BinaryReader, Int32)
Microsoft.MixedReality.SpectatorView.NetworkManager`1:OnReceive(IncomingMessage)
System.Action`1:Invoke(T)
Microsoft.MixedReality.SpectatorView.TCPConnectionManager:Update()

(Filename: C:\buildslave\unity\build\Runtime/Export/Debug/Debug.bindings.h Line: 35)

ArUcoMarkerDetector: Starting ArUco marker detection
UnityEngine.DebugLogHandler:Internal_Log(LogType, LogOption, String, Object)
UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
UnityEngine.Logger:Log(LogType, Object)
UnityEngine.Debug:Log(Object)
Microsoft.MixedReality.SpectatorView.ArUcoMarkerDetector:DebugLog(String)
Microsoft.MixedReality.SpectatorView.ArUcoMarkerDetector:StartDetecting()
Microsoft.MixedReality.SpectatorView.<EnumerateAllAssets>d__0:.ctor(Int32)
System.Array:UnsafeStore(T[], Int32, T)
Microsoft.MixedReality.SpectatorView.MarkerDetectorCoordinateService:OnDiscoverCoordinatesImplAsync(CancellationToken, Int32[])
Microsoft.MixedReality.SpectatorView.<EnumerateAllAssets>d__0:.ctor(Int32)
System.Array:UnsafeStore(T[], Int32, T)
Microsoft.MixedReality.SpectatorView.MarkerDetectorCoordinateService:OnDiscoverCoordinatesAsync(CancellationToken, Int32[])
Microsoft.Azure.SpatialAnchors.<System-Collections-IEnumerable-GetEnumerator>d__26:.ctor(Int32)
System.Array:UnsafeStore(T[], Int32, T)
Microsoft.MixedReality.SpatialAlignment.SpatialCoordinateServiceBase`1:TryDiscoverCoordinatesAsync(CancellationToken, TKey[])
Microsoft.MixedReality.SpatialAlignment.SpatialCoordinateServiceBase`1:Microsoft.MixedReality.SpatialAlignment.ISpatialCoordinateService.TryDiscoverCoordinatesAsync(CancellationToken, String[])
Microsoft.MixedReality.SpectatorView.<EnumerateAllAssets>d__0:.ctor(Int32)
System.Runtime.CompilerServices.MoveNextRunner:InvokeMoveNext(Object)
System.Threading.ContextCallback:Invoke(Object)
System.Threading.ExecutionContext:RunInternal(ExecutionContext, ContextCallback, Object, Boolean)
System.Threading.ExecutionContext:Run(ExecutionContext, ContextCallback, Object, Boolean)
System.Runtime.CompilerServices.MoveNextRunner:Run()
System.Action:Invoke()
System.Threading.Tasks.AwaitTaskContinuation:InvokeAction(Object)
System.Threading.ContextCallback:Invoke(Object)
System.Threading.Tasks.AwaitTaskContinuation:RunCallback(ContextCallback, Object, Task&)
System.Threading.Tasks.SynchronizationContextAwaitTaskContinuation:Run(Task, Boolean)
System.Threading.Tasks.Task:FinishContinuations()
System.Threading.Tasks.Task:FinishStageThree()
System.Threading.Tasks.Task`1:TrySetResult(TResult)
System.Threading.Tasks.CompleteOnInvokePromise:Invoke(Task)
System.Threading.Tasks.Task:FinishContinuations()
System.Threading.Tasks.Task:FinishStageThree()
System.Threading.Tasks.Task`1:TrySetResult(TResult)
System.Threading.Tasks.TaskCompletionSource`1:TrySetResult(TResult)
Microsoft.MixedReality.SpectatorView.LocalizationSession:OnDataReceived(BinaryReader)
Microsoft.MixedReality.SpectatorView.SpatialCoordinateSystemManager:OnParticipantDataReceived(INetworkConnection, String, BinaryReader, Int32)
Microsoft.MixedReality.SpectatorView.CommandHandler:Invoke(INetworkConnection, String, BinaryReader, Int32)
Microsoft.MixedReality.SpectatorView.CommandRegistry`1:NotifyCommand(INetworkConnection, String, BinaryReader, Int32)
Microsoft.MixedReality.SpectatorView.NetworkManager`1:OnReceive(IncomingMessage)
System.Action`1:Invoke(T)
Microsoft.MixedReality.SpectatorView.TCPConnectionManager:Update()

(Filename: C:\buildslave\unity\build\Runtime/Export/Debug/Debug.bindings.h Line: 35)

ArUcoMarkerDetector: Setting up HoloLensCamera
UnityEngine.DebugLogHandler:Internal_Log(LogType, LogOption, String, Object)
UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
UnityEngine.Logger:Log(LogType, Object)
UnityEngine.Debug:Log(Object)
Microsoft.MixedReality.SpectatorView.ArUcoMarkerDetector:DebugLog(String)
Microsoft.MixedReality.SpectatorView.ArUcoMarkerDetector:SetupCameraAsync()
Microsoft.MixedReality.SpectatorView.ArUcoMarkerDetector:StartDetecting()
Microsoft.MixedReality.SpectatorView.<EnumerateAllAssets>d__0:.ctor(Int32)
System.Array:UnsafeStore(T[], Int32, T)
Microsoft.MixedReality.SpectatorView.MarkerDetectorCoordinateService:OnDiscoverCoordinatesImplAsync(CancellationToken, Int32[])
Microsoft.MixedReality.SpectatorView.<EnumerateAllAssets>d__0:.ctor(Int32)
System.Array:UnsafeStore(T[], Int32, T)
Microsoft.MixedReality.SpectatorView.MarkerDetectorCoordinateService:OnDiscoverCoordinatesAsync(CancellationToken, Int32[])
Microsoft.Azure.SpatialAnchors.<System-Collections-IEnumerable-GetEnumerator>d__26:.ctor(Int32)
System.Array:UnsafeStore(T[], Int32, T)
Microsoft.MixedReality.SpatialAlignment.SpatialCoordinateServiceBase`1:TryDiscoverCoordinatesAsync(CancellationToken, TKey[])
Microsoft.MixedReality.SpatialAlignment.SpatialCoordinateServiceBase`1:Microsoft.MixedReality.SpatialAlignment.ISpatialCoordinateService.TryDiscoverCoordinatesAsync(CancellationToken, String[])
Microsoft.MixedReality.SpectatorView.<EnumerateAllAssets>d__0:.ctor(Int32)
System.Runtime.CompilerServices.MoveNextRunner:InvokeMoveNext(Object)
System.Threading.ContextCallback:Invoke(Object)
System.Threading.ExecutionContext:RunInternal(ExecutionContext, ContextCallback, Object, Boolean)
System.Threading.ExecutionContext:Run(ExecutionContext, ContextCallback, Object, Boolean)
System.Runtime.CompilerServices.MoveNextRunner:Run()
System.Action:Invoke()
System.Threading.Tasks.AwaitTaskContinuation:InvokeAction(Object)
System.Threading.ContextCallback:Invoke(Object)
System.Threading.Tasks.AwaitTaskContinuation:RunCallback(ContextCallback, Object, Task&)
System.Threading.Tasks.SynchronizationContextAwaitTaskContinuation:Run(Task, Boolean)
System.Threading.Tasks.Task:FinishContinuations()
System.Threading.Tasks.Task:FinishStageThree()
System.Threading.Tasks.Task`1:TrySetResult(TResult)
System.Threading.Tasks.CompleteOnInvokePromise:Invoke(Task)
System.Threading.Tasks.Task:FinishContinuations()
System.Threading.Tasks.Task:FinishStageThree()
System.Threading.Tasks.Task`1:TrySetResult(TResult)
System.Threading.Tasks.TaskCompletionSource`1:TrySetResult(TResult)
Microsoft.MixedReality.SpectatorView.LocalizationSession:OnDataReceived(BinaryReader)
Microsoft.MixedReality.SpectatorView.SpatialCoordinateSystemManager:OnParticipantDataReceived(INetworkConnection, String, BinaryReader, Int32)
Microsoft.MixedReality.SpectatorView.CommandHandler:Invoke(INetworkConnection, String, BinaryReader, Int32)
Microsoft.MixedReality.SpectatorView.CommandRegistry`1:NotifyCommand(INetworkConnection, String, BinaryReader, Int32)
Microsoft.MixedReality.SpectatorView.NetworkManager`1:OnReceive(IncomingMessage)
System.Action`1:Invoke(T)
Microsoft.MixedReality.SpectatorView.TCPConnectionManager:Update()

(Filename: C:\buildslave\unity\build\Runtime/Export/Debug/Debug.bindings.h Line: 35)

MarkerDetector: Marker with id: 1 wasn't found yet.
UnityEngine.DebugLogHandler:Internal_Log(LogType, LogOption, String, Object)
UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
UnityEngine.Logger:Log(LogType, Object)
UnityEngine.Debug:Log(Object)
Microsoft.MixedReality.SpectatorView.MarkerDetectorCoordinateService:DebugLog(String)
Microsoft.MixedReality.SpectatorView.<EnumerateAllAssets>d__0:.ctor(Int32)
System.Array:UnsafeStore(T[], Int32, T)
Microsoft.MixedReality.SpectatorView.MarkerDetectorCoordinateService:OnDiscoverCoordinatesImplAsync(CancellationToken, Int32[])
Microsoft.MixedReality.SpectatorView.<EnumerateAllAssets>d__0:.ctor(Int32)
System.Array:UnsafeStore(T[], Int32, T)
Microsoft.MixedReality.SpectatorView.MarkerDetectorCoordinateService:OnDiscoverCoordinatesAsync(CancellationToken, Int32[])
Microsoft.Azure.SpatialAnchors.<System-Collections-IEnumerable-GetEnumerator>d__26:.ctor(Int32)
System.Array:UnsafeStore(T[], Int32, T)
Microsoft.MixedReality.SpatialAlignment.SpatialCoordinateServiceBase`1:TryDiscoverCoordinatesAsync(CancellationToken, TKey[])
Microsoft.MixedReality.SpatialAlignment.SpatialCoordinateServiceBase`1:Microsoft.MixedReality.SpatialAlignment.ISpatialCoordinateService.TryDiscoverCoordinatesAsync(CancellationToken, String[])
Microsoft.MixedReality.SpectatorView.<EnumerateAllAssets>d__0:.ctor(Int32)
System.Runtime.CompilerServices.MoveNextRunner:InvokeMoveNext(Object)
System.Threading.ContextCallback:Invoke(Object)
System.Threading.ExecutionContext:RunInternal(ExecutionContext, ContextCallback, Object, Boolean)
System.Threading.ExecutionContext:Run(ExecutionContext, ContextCallback, Object, Boolean)
System.Runtime.CompilerServices.MoveNextRunner:Run()
System.Action:Invoke()
System.Threading.Tasks.AwaitTaskContinuation:InvokeAction(Object)
System.Threading.ContextCallback:Invoke(Object)
System.Threading.Tasks.AwaitTaskContinuation:RunCallback(ContextCallback, Object, Task&)
System.Threading.Tasks.SynchronizationContextAwaitTaskContinuation:Run(Task, Boolean)
System.Threading.Tasks.Task:FinishContinuations()
System.Threading.Tasks.Task:FinishStageThree()
System.Threading.Tasks.Task`1:TrySetResult(TResult)
System.Threading.Tasks.CompleteOnInvokePromise:Invoke(Task)
System.Threading.Tasks.Task:FinishContinuations()
System.Threading.Tasks.Task:FinishStageThree()
System.Threading.Tasks.Task`1:TrySetResult(TResult)
System.Threading.Tasks.TaskCompletionSource`1:TrySetResult(TResult)
Microsoft.MixedReality.SpectatorView.LocalizationSession:OnDataReceived(BinaryReader)
Microsoft.MixedReality.SpectatorView.SpatialCoordinateSystemManager:OnParticipantDataReceived(INetworkConnection, String, BinaryReader, Int32)
Microsoft.MixedReality.SpectatorView.CommandHandler:Invoke(INetworkConnection, String, BinaryReader, Int32)
Microsoft.MixedReality.SpectatorView.CommandRegistry`1:NotifyCommand(INetworkConnection, String, BinaryReader, Int32)
Microsoft.MixedReality.SpectatorView.NetworkManager`1:OnReceive(IncomingMessage)
System.Action`1:Invoke(T)
Microsoft.MixedReality.SpectatorView.TCPConnectionManager:Update()

(Filename: C:\buildslave\unity\build\Runtime/Export/Debug/Debug.bindings.h Line: 35)

MarkerDetector: Waiting another second
UnityEngine.DebugLogHandler:Internal_Log(LogType, LogOption, String, Object)
UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
UnityEngine.Logger:Log(LogType, Object)
UnityEngine.Debug:Log(Object)
Microsoft.MixedReality.SpectatorView.MarkerDetectorCoordinateService:DebugLog(String)
Microsoft.MixedReality.SpectatorView.<EnumerateAllAssets>d__0:.ctor(Int32)
System.Array:UnsafeStore(T[], Int32, T)
Microsoft.MixedReality.SpectatorView.MarkerDetectorCoordinateService:OnDiscoverCoordinatesImplAsync(CancellationToken, Int32[])
Microsoft.MixedReality.SpectatorView.<EnumerateAllAssets>d__0:.ctor(Int32)
System.Array:UnsafeStore(T[], Int32, T)
Microsoft.MixedReality.SpectatorView.MarkerDetectorCoordinateService:OnDiscoverCoordinatesAsync(CancellationToken, Int32[])
Microsoft.Azure.SpatialAnchors.<System-Collections-IEnumerable-GetEnumerator>d__26:.ctor(Int32)
System.Array:UnsafeStore(T[], Int32, T)
Microsoft.MixedReality.SpatialAlignment.SpatialCoordinateServiceBase`1:TryDiscoverCoordinatesAsync(CancellationToken, TKey[])
Microsoft.MixedReality.SpatialAlignment.SpatialCoordinateServiceBase`1:Microsoft.MixedReality.SpatialAlignment.ISpatialCoordinateService.TryDiscoverCoordinatesAsync(CancellationToken, String[])
Microsoft.MixedReality.SpectatorView.<EnumerateAllAssets>d__0:.ctor(Int32)
System.Runtime.CompilerServices.MoveNextRunner:InvokeMoveNext(Object)
System.Threading.ContextCallback:Invoke(Object)
System.Threading.ExecutionContext:RunInternal(ExecutionContext, ContextCallback, Object, Boolean)
System.Threading.ExecutionContext:Run(ExecutionContext, ContextCallback, Object, Boolean)
System.Runtime.CompilerServices.MoveNextRunner:Run()
System.Action:Invoke()
System.Threading.Tasks.AwaitTaskContinuation:InvokeAction(Object)
System.Threading.ContextCallback:Invoke(Object)
System.Threading.Tasks.AwaitTaskContinuation:RunCallback(ContextCallback, Object, Task&)
System.Threading.Tasks.SynchronizationContextAwaitTaskContinuation:Run(Task, Boolean)
System.Threading.Tasks.Task:FinishContinuations()
System.Threading.Tasks.Task:FinishStageThree()
System.Threading.Tasks.Task`1:TrySetResult(TResult)
System.Threading.Tasks.CompleteOnInvokePromise:Invoke(Task)
System.Threading.Tasks.Task:FinishContinuations()
System.Threading.Tasks.Task:FinishStageThree()
System.Threading.Tasks.Task`1:TrySetResult(TResult)
System.Threading.Tasks.TaskCompletionSource`1:TrySetResult(TResult)
Microsoft.MixedReality.SpectatorView.LocalizationSession:OnDataReceived(BinaryReader)
Microsoft.MixedReality.SpectatorView.SpatialCoordinateSystemManager:OnParticipantDataReceived(INetworkConnection, String, BinaryReader, Int32)
Microsoft.MixedReality.SpectatorView.CommandHandler:Invoke(INetworkConnection, String, BinaryReader, Int32)
Microsoft.MixedReality.SpectatorView.CommandRegistry`1:NotifyCommand(INetworkConnection, String, BinaryReader, Int32)
Microsoft.MixedReality.SpectatorView.NetworkManager`1:OnReceive(IncomingMessage)
System.Action`1:Invoke(T)
Microsoft.MixedReality.SpectatorView.TCPConnectionManager:Update()

(Filename: C:\buildslave\unity\build\Runtime/Export/Debug/Debug.bindings.h Line: 35)

ArUcoMarkerDetector: HoloLensCamera initialized
UnityEngine.DebugLogHandler:Internal_Log(LogType, LogOption, String, Object)
UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
UnityEngine.Logger:Log(LogType, Object)
UnityEngine.Debug:Log(Object)
Microsoft.MixedReality.SpectatorView.ArUcoMarkerDetector:DebugLog(String)
Microsoft.MixedReality.SpectatorView.ArUcoMarkerDetector:CameraInitialized(HoloLensCamera, Boolean)
Microsoft.MixedReality.PhotoCapture.OnCameraInitializedHandler:Invoke(HoloLensCamera, Boolean)
Microsoft.MixedReality.PhotoCapture.StreamSelector:set_StreamDescriptions(List`1)
System.Runtime.CompilerServices.MoveNextRunner:InvokeMoveNext(Object)
System.Threading.ContextCallback:Invoke(Object)
System.Threading.ExecutionContext:RunInternal(ExecutionContext, ContextCallback, Object, Boolean)
System.Threading.ExecutionContext:Run(ExecutionContext, ContextCallback, Object, Boolean)
System.Runtime.CompilerServices.MoveNextRunner:Run()
System.Action:Invoke()
System.Threading.Tasks.<>c:<.cctor>b__7_0(Object)
System.Threading.SendOrPostCallback:Invoke(Object)
UnityEngine.WSA.WindowSizeChanged:.ctor(Object, IntPtr)
UnityEngine.UnitySynchronizationContext:Exec()
UnityEngine.UnitySynchronizationContext:ExecuteTasks()

(Filename: C:\buildslave\unity\build\Runtime/Export/Debug/Debug.bindings.h Line: 35)

Exception thrown at 0x76EA3072 (KernelBase.dll) in MixedRealityWebRTCUnityDemo.exe: 0x8001010E: The application called an interface that was marshalled for a different thread.
Exception thrown at 0x76EA3072 in MixedRealityWebRTCUnityDemo.exe: Microsoft C++ exception: Il2CppExceptionWrapper at memory location 0x02C0DBB4.
Exception thrown at 0x76EA3072 in MixedRealityWebRTCUnityDemo.exe: Microsoft C++ exception: Il2CppExceptionWrapper at memory location 0x02C0ED00.
Exception: Exception of type 'System.Exception' was thrown.
  at Windows.Media.Capture.MediaCapture..ctor () [0x00000] in <00000000000000000000000000000000>:0 
  at Microsoft.MixedReality.PhotoCapture.StreamSelector.set_StreamDescriptions (System.Collections.Generic.List`1[T] value) [0x00000] in <00000000000000000000000000000000>:0 
  at Microsoft.MixedReality.SpectatorView.AssetService.LookupAssetCache[T] () [0x00000] in <00000000000000000000000000000000>:0 
  at Microsoft.MixedReality.PhotoCapture.HoloLensCamera.Start (Microsoft.MixedReality.PhotoCapture.StreamDescription streamDesc) [0x00000] in <00000000000000000000000000000000>:0 
  at Microsoft.MixedReality.SpectatorView.ArUcoMarkerDetector.CameraInitialized (Microsoft.MixedReality.PhotoCapture.HoloLensCamera sender, System.Boolean initializeSuccessful) [0x00000] in <00000000000000000000000000000000>:0 
  at Microsoft.MixedReality.PhotoCapture.OnCameraInitializedHandler.Invoke (Microsoft.MixedReality.PhotoCapture.HoloLensCamera sender, System.Boolean initializeSuccessful) [0x00000] in <00000000000000000000000000000000>:0 
  at Microsoft.MixedReality.PhotoCapture.StreamSelector.set_StreamDescriptions (System.Collections.Generic.List`1[T] value) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.InvokeMoveNext (System.Object stateMachine) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Threading.ContextCallback.Invoke (System.Object state) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () [0x00000] in <00000000000000000000000000000000>:0 
  at System.Action.Invoke () [0x00000] in <00000000000000000000000000000000>:0 
  at System.Threading.Tasks.SynchronizationContextAwaitTaskContinuation+<>c.<.cctor>b__7_0 (System.Object state) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Threading.SendOrPostCallback.Invoke (System.Object state) [0x00000] in <00000000000000000000000000000000>:0 
  at UnityEngine.WSA.WindowSizeChanged..ctor (System.Object object, System.IntPtr method) [0x00000] in <00000000000000000000000000000000>:0 
  at UnityEngine.UnitySynchronizationContext.Exec () [0x00000] in <00000000000000000000000000000000>:0 
  at UnityEngine.UnitySynchronizationContext.ExecuteTasks () [0x00000] in <00000000000000000000000000000000>:0 
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x00000] in <00000000000000000000000000000000>:0 
  at System.Runtime.CompilerServices.AsyncMethodBuilderCore+<>c.<ThrowAsync>b__6_0 (System.Object state) [0x00000] in <00000000000000000000000000000000>:0 
  at System.Threading.SendOrPostCallback.Invoke (System.Object state) [0x00000] in <00000000000000000000000000000000>:0 
  at UnityEngine.WSA.WindowSizeChanged..ctor (System.Object object, System.IntPtr method) [0x00000] in <00000000000000000000000000000000>:0 
  at UnityEngine.UnitySynchronizationContext.Exec () [0x00000] in <00000000000000000000000000000000>:0 
  at UnityEngine.UnitySynchronizationContext.ExecuteTasks () [0x00000] in <00000000000000000000000000000000>:0 
UnityEngine.DebugLogHandler:Internal_LogException(Exception, Object)
UnityEngine.DebugLogHandler:LogException(Exception, Object)
UnityEngine.Logger:LogException(Exception, Object)
UnityEngine.Debug:LogException(Exception)
UnityEngine.WSA.WindowSizeChanged:.ctor(Object, IntPtr)
UnityEngine.UnitySynchronizationContext:Exec()
UnityEngine.UnitySynchronizationContext:ExecuteTasks()

aleneum commented 4 years ago

Using SpectatorView and WebRTC in different scenes has the same effect.

chrisfromwork commented 4 years ago

Hm, this does seem like strange behavior. I would hope that even if we were doing something in Spectator View incorrectly that WebRTC would successfully clean things up.

chrisfromwork commented 4 years ago

I'll make sure this is drawn to the attention of WebRTC owners and will see if this is a known issue.

aleneum commented 4 years ago

Hi @chrisfromwork,

I also filed an issue at the WebRTC project as suggested by Jesse McCulloch in the #specator_view Slack chat (https://github.com/microsoft/MixedReality-WebRTC/issues/175).

@djee-ms mentioned the following

Yes I am aware that at the moment the MediaCapture is locked by the local video track. I will have to talk with @chrisfromwork about this to understand the details and how we can share the MediaCapture between the two projects.

From what I can tell (with my limited knowledge about how things work in Unity/C#/UWP), it seems that SpectatorView is correctly disposing HoloLensCamera and also its MediaCapture reference. There should also be enough time (10s+) to free the ressources before the next allocation happens with WebRTC. WebRTC and SpectatorView do initiate MediaCapture with different parameters but that should not be an issue, should it... Idk... does the garbage collector has to be triggered before the reference is actually released? Are there other objects (in addition to HoloLensCamera) that might have a reference to the MediaCapture or some of its resources (the last received image) that prevent proper releasing? Or is the issue maybe even Unity-related or happens in some other unmanaged code?

Anyway, thank you for having a look at it 👍 !

chrisfromwork commented 4 years ago

We think we may have identified a potential issue. MediaCapture should be initialized on the UWP UI Thread, which isn't the same as the Unity update thread. It looks like in ArUcoMarkerDetector/HoloLensCamera we do not initialize on the UWP UI Thread, while in WebRTC we do.

An untested workaround is to change the HoloLensCamera code to execute mediaCapture calls on the UWP UI Thread. To do this, you can do something like the following (code from WebRTC). You will need to replace the calls to RequestAccessAndInitAsync with the code that starts/initializes/disposes the mediaCapture class.

if UNITY_WSA && !UNITY_EDITOR

        if (UnityEngine.WSA.Application.RunningOnUIThread())

endif

        {
            return RequestAccessAndInitAsync(token);
        }

if UNITY_WSA && !UNITY_EDITOR

        else
        {
            UnityEngine.WSA.Application.InvokeOnUIThread(() => RequestAccessAndInitAsync(token), waitUntilDone: true);
            return Task.CompletedTask;
        }

endif

We will look into fixing this, but it will likely be delayed based on other issues/projects that are higher priotity/going on in parallel.

aleneum commented 4 years ago

Hi @chrisfromwork,

thank you for your investigation.

An untested workaround is to change the HoloLensCamera code to execute mediaCapture calls on the UWP UI Thread.

I gave that a try by wrapping sender.Start(streamDesc) in ArUcoMarkerDetector.CameraInitialized:

        private void CameraInitialized(HoloLensCamera sender, bool initializeSuccessful)
        {
            DebugLog("HoloLensCamera initialized");
            StreamDescription streamDesc = sender.StreamSelector.Select(StreamCompare.EqualTo, 1280, 720).StreamDescriptions[0];
#if UNITY_WSA && !UNITY_EDITOR
            if (UnityEngine.WSA.Application.RunningOnUIThread())
#endif
            {
                 sender.Start(streamDesc);
            }
#if UNITY_WSA && !UNITY_EDITOR
            else
            {
                UnityEngine.WSA.Application.InvokeOnUIThread(() => sender.Start(streamDesc), waitUntilDone: false);
            }
#endif
        }

I set waitUntilDone = false since Start is not async and probably deadlocks otherwise. I added some good old Debug.Log in HoloLensCamera.Start and could verify that it is executed successfully. However, grabbing frames fails. VideoDeviceController.get_IsoSpeedControl causes a COMException:


HoloLensCamera started
UnityEngine.DebugLogHandler:Internal_Log(LogType, LogOption, String, Object)
UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
UnityEngine.Logger:Log(LogType, Object)
UnityEngine.Debug:Log(Object)
Microsoft.MixedReality.PhotoCapture.StreamSelector:set_StreamDescriptions(List`1)
System.Runtime.CompilerServices.MoveNextRunner:InvokeMoveNext(Object)
System.Threading.ContextCallback:Invoke(Object)
System.Threading.ExecutionContext:RunInternal(ExecutionContext, ContextCallback, Object, Boolean)
System.Threading.ExecutionContext:Run(ExecutionContext, ContextCallback, Object, Boolean)
System.Runtime.CompilerServices.MoveNextRunner:Run()
System.Action:Invoke()
System.Threading.Tasks.<>c:<.cctor>b__7_0(Object)
System.Threading.SendOrPostCallback:Invoke(Object)
System.Threading.InvocationContext:Invoke()
System.Threading.OSSpecificSynchronizationContext:InvocationEntry(IntPtr)

(Filename: C:\buildslave\unity\build\Runtime/Export/Debug/Debug.bindings.h Line: 35)

ArUcoMarkerDetector: HoloLensCamera successfully started
UnityEngine.DebugLogHandler:Internal_Log(LogType, LogOption, String, Object)
UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
UnityEngine.Logger:Log(LogType, Object)
UnityEngine.Debug:Log(Object)
Microsoft.MixedReality.SpectatorView.ArUcoMarkerDetector:DebugLog(String)
Microsoft.MixedReality.SpectatorView.ArUcoMarkerDetector:CameraStarted(HoloLensCamera, Boolean)
Microsoft.MixedReality.PhotoCapture.OnCameraStartedHandler:Invoke(HoloLensCamera, Boolean)
Microsoft.MixedReality.PhotoCapture.StreamSelector:set_StreamDescriptions(List`1)
System.Runtime.CompilerServices.MoveNextRunner:InvokeMoveNext(Object)
System.Threading.ContextCallback:Invoke(Object)
System.Threading.ExecutionContext:RunInternal(ExecutionContext, ContextCallback, Object, Boolean)
System.Threading.ExecutionContext:Run(ExecutionContext, ContextCallback, Object, Boolean)
System.Runtime.CompilerServices.MoveNextRunner:Run()
System.Action:Invoke()
System.Threading.Tasks.<>c:<.cctor>b__7_0(Object)
System.Threading.SendOrPostCallback:Invoke(Object)
System.Threading.InvocationContext:Invoke()
System.Threading.OSSpecificSynchronizationContext:InvocationEntry(IntPtr)

(Filename: C:\buildslave\unity\build\Runtime/Export/Debug/Debug.bindings.h Line: 35)

Exception thrown at 0x777C3072 in MixedRealityWebRTCUnityDemo.exe: Microsoft C++ exception: Il2CppExceptionWrapper at memory location 0x2E9FF404.
Exception thrown at 0x777C3072 (KernelBase.dll) in MixedRealityWebRTCUnityDemo.exe: WinRT originate error - 0x8001010E : 'System.Runtime.InteropServices.COMException (0x8001010E): Error HRESULT E_FAIL has been returned from a call to a COM component.
  at Windows.Media.Devices.VideoDeviceController.get_IsoSpeedControl () [0x00000] in <00000000000000000000000000000000>:0 
  at Microsoft.MixedReality.PhotoCapture.HoloLensCamera.get_Gain () [0x00000] in <00000000000000000000000000000000>:0 
  at Microsoft.MixedReality.PhotoCapture.HoloLensCamera.GetFrameFromMediaFrameReader (Windows.Media.Capture.Frames.MediaFrameReader frameR'.
Exception thrown at 0x777C3072 (KernelBase.dll) in MixedRealityWebRTCUnityDemo.exe: 0x40080202: WinRT transform error (parameters: 0x8001010E, 0x00000000, 0x00000027, 0x2E9FF0DC).
MarkerDetector: Marker with id: 1 wasn't found yet.
UnityEngine.DebugLogHandler:Internal_Log(LogType, LogOption, String, Object)
UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
UnityEngine.Logger:Log(LogType, Object)
UnityEngine.Debug:Log(Object)
Microsoft.MixedReality.SpectatorView.MarkerDetectorCoordinateService:DebugLog(String)
Microsoft.MixedReality.SpectatorView.<EnumerateAllAssets>d__0:.ctor(Int32)
System.Runtime.CompilerServices.MoveNextRunner:InvokeMoveNext(Object)
System.Threading.ContextCallback:Invoke(Object)
System.Threading.ExecutionContext:RunInternal(ExecutionContext, ContextCallback, Object, Boolean)
System.Threading.ExecutionContext:Run(ExecutionContext, ContextCallback, Object, Boolean)
System.Runtime.CompilerServices.MoveNextRunner:Run()
System.Action:Invoke()
System.Threading.Tasks.<>c:<.cctor>b__7_0(Object)
System.Threading.SendOrPostCallback:Invoke(Object)
UnityEngine.WSA.WindowSizeChanged:.ctor(Object, IntPtr)
UnityEngine.UnitySynchronizationContext:Exec()
UnityEngine.UnitySynchronizationContext:ExecuteTasks()

(Filename: C:\buildslave\unity\build\Runtime/Export/Debug/Debug.bindings.h Line: 35)

Exception thrown at 0x777C3072 in MixedRealityWebRTCUnityDemo.exe: Microsoft C++ exception: Il2CppExceptionWrapper at memory location 0x2E9FF3F4.
MarkerDetector: Waiting another second
UnityEngine.DebugLogHandler:Internal_Log(LogType, LogOption, String, Object)
UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
UnityEngine.Logger:Log(LogType, Object)
UnityEngine.Debug:Log(Object)
Microsoft.MixedReality.SpectatorView.MarkerDetectorCoordinateService:DebugLog(String)
Microsoft.MixedReality.SpectatorView.<EnumerateAllAssets>d__0:.ctor(Int32)
System.Runtime.CompilerServices.MoveNextRunner:InvokeMoveNext(Object)
System.Threading.ContextCallback:Invoke(Object)
System.Threading.ExecutionContext:RunInternal(ExecutionContext, ContextCallback, Object, Boolean)
System.Threading.ExecutionContext:Run(ExecutionContext, ContextCallback, Object, Boolean)
System.Runtime.CompilerServices.MoveNextRunner:Run()
System.Action:Invoke()
System.Threading.Tasks.<>c:<.cctor>b__7_0(Object)
System.Threading.SendOrPostCallback:Invoke(Object)
UnityEngine.WSA.WindowSizeChanged:.ctor(Object, IntPtr)
UnityEngine.UnitySynchronizationContext:Exec()
UnityEngine.UnitySynchronizationContext:ExecuteTasks()

(Filename: C:\buildslave\unity\build\Runtime/Export/Debug/Debug.bindings.h Line: 35)

Exception thrown at 0x777C3072 (KernelBase.dll) in MixedRealityWebRTCUnityDemo.exe: WinRT originate error - 0x8001010E : 'System.Runtime.InteropServices.COMException (0x8001010E): Error HRESULT E_FAIL has been returned from a call to a COM component.
  at Windows.Media.Devices.VideoDeviceController.get_IsoSpeedControl () [0x00000] in <00000000000000000000000000000000>:0 
  at Microsoft.MixedReality.PhotoCapture.HoloLensCamera.get_Gain () [0x00000] in <00000000000000000000000000000000>:0 
  at Microsoft.MixedReality.PhotoCapture.HoloLensCamera.GetFrameFromMediaFrameReader (Windows.Media.Capture.Frames.MediaFrameReader frameR'.
Exception thrown at 0x777C3072 (KernelBase.dll) in MixedRealityWebRTCUnityDemo.exe: 0x40080202: WinRT transform error (parameters: 0x8001010E, 0x00000000, 0x00000027, 0x2E9FF0C8).
Exception thrown at 0x777C3072 in MixedRealityWebRTCUnityDemo.exe: Microsoft C++ exception: Il2CppExceptionWrapper at memory location 0x2E9FF3F4.
Exception thrown at 0x777C3072 (KernelBase.dll) in MixedRealityWebRTCUnityDemo.exe: WinRT originate error - 0x8001010E : 'System.Runtime.InteropServices.COMException (0x8001010E): Error HRESULT E_FAIL has been returned from a call to a COM component.
  at Windows.Media.Devices.VideoDeviceController.get_IsoSpeedControl () [0x00000] in <00000000000000000000000000000000>:0 
  at Microsoft.MixedReality.PhotoCapture.HoloLensCamera.get_Gain () [0x00000] in <00000000000000000000000000000000>:0 
  at Microsoft.MixedReality.PhotoCapture.HoloLensCamera.GetFrameFromMediaFrameReader (Windows.Media.Capture.Frames.MediaFrameReader frameR'.
Exception thrown at 0x777C3072 (KernelBase.dll) in MixedRealityWebRTCUnityDemo.exe: 0x40080202: WinRT transform error (parameters: 0x8001010E, 0x00000000, 0x00000027, 0x2E9FF0C8).
MarkerDetector: Marker with id: 1 wasn't found yet.
UnityEngine.DebugLogHandler:Internal_Log(LogType, LogOption, String, Object)
UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
UnityEngine.Logger:Log(LogType, Object)
UnityEngine.Debug:Log(Object)
Microsoft.MixedReality.SpectatorView.MarkerDetectorCoordinateService:DebugLog(String)
Microsoft.MixedReality.SpectatorView.<EnumerateAllAssets>d__0:.ctor(Int32)
System.Runtime.CompilerServices.MoveNextRunner:InvokeMoveNext(Object)
System.Threading.ContextCallback:Invoke(Object)
System.Threading.ExecutionContext:RunInternal(ExecutionContext, ContextCallback, Object, Boolean)
System.Threading.ExecutionContext:Run(ExecutionContext, ContextCallback, Object, Boolean)
System.Runtime.CompilerServices.MoveNextRunner:Run()
System.Action:Invoke()
System.Threading.Tasks.<>c:<.cctor>b__7_0(Object)
System.Threading.SendOrPostCallback:Invoke(Object)
UnityEngine.WSA.WindowSizeChanged:.ctor(Object, IntPtr)
UnityEngine.UnitySynchronizationContext:Exec()
UnityEngine.UnitySynchronizationContext:ExecuteTasks()

(Filename: C:\buildslave\unity\build\Runtime/Export/Debug/Debug.bindings.h Line: 35)

MarkerDetector: Waiting another second
UnityEngine.DebugLogHandler:Internal_Log(LogType, LogOption, String, Object)
UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
UnityEngine.Logger:Log(LogType, Object)
UnityEngine.Debug:Log(Object)
Microsoft.MixedReality.SpectatorView.MarkerDetectorCoordinateService:DebugLog(String)
Microsoft.MixedReality.SpectatorView.<EnumerateAllAssets>d__0:.ctor(Int32)
System.Runtime.CompilerServices.MoveNextRunner:InvokeMoveNext(Object)
System.Threading.ContextCallback:Invoke(Object)
System.Threading.ExecutionContext:RunInternal(ExecutionContext, ContextCallback, Object, Boolean)
System.Threading.ExecutionContext:Run(ExecutionContext, ContextCallback, Object, Boolean)
System.Runtime.CompilerServices.MoveNextRunner:Run()
System.Action:Invoke()
System.Threading.Tasks.<>c:<.cctor>b__7_0(Object)
System.Threading.SendOrPostCallback:Invoke(Object)
UnityEngine.WSA.WindowSizeChanged:.ctor(Object, IntPtr)
UnityEngine.UnitySynchronizationContext:Exec()
UnityEngine.UnitySynchronizationContext:ExecuteTasks()

(Filename: C:\buildslave\unity\build\Runtime/Export/Debug/Debug.bindings.h Line: 35)

microsoft / MixedReality-SpectatorView

Initializing MediaCapture fails when WebRTC has been used before #322

System

Scenario 1: Spectator View first, WebRTC second

Initializing `HoloLensCamera`

Releasing `HoloLensCamera`

Initializing `MediaCapture` in `PeerConnection`

Scenario 2: WebRTC first. SpectatorView second

Initializing `MediaCapture` in `PeerConnection`

Destroying WebRTC GameObject (or disabling it)

Initializing `HoloLensCamera`

Procedure

Result

if UNITY_WSA && !UNITY_EDITOR

endif

if UNITY_WSA && !UNITY_EDITOR

endif

microsoft / MixedReality-SpectatorView

Initializing MediaCapture fails when WebRTC has been used before #322

System

Scenario 1: Spectator View first, WebRTC second

Initializing HoloLensCamera

Releasing HoloLensCamera

Initializing MediaCapture in PeerConnection

Scenario 2: WebRTC first. SpectatorView second

Initializing MediaCapture in PeerConnection

Destroying WebRTC GameObject (or disabling it)

Initializing HoloLensCamera

Procedure

Result

if UNITY_WSA && !UNITY_EDITOR

endif

if UNITY_WSA && !UNITY_EDITOR

endif

Initializing `HoloLensCamera`

Releasing `HoloLensCamera`

Initializing `MediaCapture` in `PeerConnection`

Initializing `MediaCapture` in `PeerConnection`

Initializing `HoloLensCamera`