Closed AAvrilML closed 5 years ago
Magic Leap isn't a platform we properly support at the moment, although it uses Android I think it's a heavily modified version which won't load our binary dependencies (audio preprocessor and opus codec). However, I would expect that to throw some kind of error if it was the problem. Could you turn up the logging settings for Dissonance (Window > Dissonance > Diagnostic Settings) to Debug
for all categories and see me a log (paste it here or email to martin@placeholder-software.co.uk). Thanks :)
Hi @martindevans has there been any updates or plans to support Magic Leap? Thanks!
We don't have any concrete plans for magic leap support at the moment because we don't have the actual device to test on and it's not been very highly requested. As it stands we'd need someone who is willing to work with us testing on the actual device.
@martindevans i just tried my dissonance project on the magic leap here a bit of a log from booting the start up of dissonance running on magic leap
Unity version: Unity 2018.1.6f1-MLTP7 (64-bit) Dissonance: HLAPI v6.2.6 (2018-10-31)
`01-09 14:08:56.188 2142 1 D Unity : Looking for Plugin opus -> /package/bin
01-09 14:08:56.188 2142 1 D Unity : Looking for Plugin opus -> /system/lib64
01-09 14:08:56.194 2142 1 E Unity : EntryPointNotFoundException: Unable to find an entry point named 'dissonance_opus_encoder_ctl_in' in 'opus'.
01-09 14:08:56.194 2142 1 E Unity : at UnityEngine.JsonUtility.FromJson (System.String json, System.Type type) [0x00000] in <filename unknown>:0
01-09 14:08:56.194 2142 1 E Unity : at UnityEngine.JsonUtility.FromJson (System.String json, System.Type type) [0x00000] in <filename unknown>:0
01-09 14:08:56.194 2142 1 E Unity : at Dissonance.Audio.Codecs.Opus.OpusNative+OpusNativeMethods.dissonance_opus_encoder_ctl_in (IntPtr st, Ctl request, Int32 value) [0x00000] in <filename unknown>:0
01-09 14:08:56.194 2142 1 E Unity : at Dissonance.Audio.Codecs.Opus.OpusNative+OpusEncoder.OpusCtlIn (Ctl ctl, Int32 value) [0x00000] in <filename unknown>:0
01-09 14:08:56.194 2142 1 E Unity : at Dissonance.Audio.Codecs.Opus.OpusNative+OpusEncoder.set_EnableForwardErrorCorrection (Boolean value) [0x00000] in <filename unknown>:0
01-09 14:08:56.194 2142 1 E Unity : at Dissonance.Audio.Codecs.Opus.OpusEncoder..ctor (AudioQuality quality, FrameSize frameSize, Boolean fec) [0x00000] in <filename unknown>:0
01-09 14:08:56.194 2142 1 E Unity : at Dissonance.CodecSettingsLoader.CreateEncoder (AudioQuality quality, FrameSize frameSize, Boolean fec) [0x00000] in <filename unknown>:0
01-09 14:08:56.194 2142 1 E Unity : at Dissonance.CodecSettingsLoader.Generate () [0x00000] in <filename unknown>:0
01-09 14:08:56.194 2142 1 E Unity : at Dissonance.CodecSettingsLoader.get_Config () [0x00000] in <filename unknown>:0
01-09 14:08:56.194 2142 1 E Unity : at Dissonance.DissonanceComms.Start () [0x00000] in <filename unknown>:0
01-09 14:08:56.194 2142 1 E Unity : at System.Collections.Generic.IList`1[T].get_Item (Int32 index) [0x00000] in <filename unknown>:0
01-09 14:08:56.194 2142 1 E Unity : at UnityEngine.CSSLayout.Native..cctor () [0x00000] in <filename unknown>:0
01-09 14:08:56.194 2142 1 E Unity : System.Collections.Generic.IList`1:get_Item(Int32)
01-09 14:08:56.194 2142 1 E Unity : UnityEngine.CSSLayout.Native:.cctor()
01-09 14:08:56.194 2142 1 E Unity :
01-09 14:08:56.194 2142 1 E Unity : (Filename: currently not available on il2cpp Line: -1)`
and from shutdown
01-09 15:16:37.852 2685 4 D Unity : UnityPerf: lifecycle on pause called at 36.352622 seconds
01-09 15:16:37.858 2685 1 E Unity : Error: MagicLeapDevice.Update failed to get valid snapshot. Snapshot is NULL
01-09 15:16:37.858 2685 1 E Unity : System.Diagnostics.StackTrace:.ctor(Int32, Boolean)
01-09 15:16:37.858 2685 1 E Unity : UnityEngine.StackTraceUtility:ExtractStackTrace()
01-09 15:16:37.858 2685 1 E Unity : UnityEngine.CSSLayout.Native:.cctor()
01-09 15:16:37.858 2685 1 E Unity : UnityEngine.DebugLogHandler:Internal_Log(LogType, String, Object)
01-09 15:16:37.858 2685 1 E Unity : UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
01-09 15:16:37.858 2685 1 E Unity : UnityEngine.Logger:Log(LogType, Object)
01-09 15:16:37.858 2685 1 E Unity : UnityEngine.Debug:LogError(Object)
01-09 15:16:37.858 2685 1 E Unity : UnityEngine.XR.MagicLeap.MLPluginLog:Error(Object)
01-09 15:16:37.858 2685 1 E Unity : UnityEngine.XR.MagicLeap.MagicLeapDevice:Update()
01-09 15:16:37.858 2685 1 E Unity : HandyCollections.Heap.IMinHeap`1:IndexOf(Predicate`1)
01-09 15:16:37.858 2685 1 E Unity : UnityEngine.CSSLayout.Native:.cctor()
01-09 15:16:37.858 2685 1 E Unity :
01-09 15:16:37.858 2685 1 E Unity : (Filename: C:/buildslave/unity/build\Runtime/Export/Debug.bindings.h Line: 43)
01-09 15:16:37.858 2685 1 E Unity :
01-09 15:16:37.864 2685 1 I Unity : Network layer will still work after application suspends
01-09 15:16:37.864 2685 1 I Unity : System.Diagnostics.StackTrace:.ctor(Int32, Boolean)
01-09 15:16:37.864 2685 1 I Unity : UnityEngine.StackTraceUtility:ExtractStackTrace()
01-09 15:16:37.864 2685 1 I Unity : UnityEngine.CSSLayout.Native:.cctor()
01-09 15:16:37.864 2685 1 I Unity :
01-09 15:16:37.864 2685 1 I Unity : [C:/buildslave/unity/build/Runtime/Networking/UNETManager.cpp line 220]
01-09 15:16:37.864 2685 1 I Unity : (Filename: C:/buildslave/unity/build/Runtime/Networking/UNETManager.cpp Line: 220)
01-09 15:16:37.864 2685 1 I Unity :
01-09 15:16:41.974 2685 35 D Unity : UnityPerf: lifecycle on stop called at 40.474970 seconds
01-09 15:16:42.022 2685 1 I Unity : DEBUG [Dissonance:Core] (06:16:42.001) PlayerTrackerManager: Removed unlinked state tracker for 'd3d2e5c0-9dc7-4f7b-bd00-44142ed237c1' (because RemoveTracker called)
01-09 15:16:42.022 2685 1 I Unity : System.Diagnostics.StackTrace:.ctor(Int32, Boolean)
01-09 15:16:42.022 2685 1 I Unity : UnityEngine.StackTraceUtility:ExtractStackTrace()
01-09 15:16:42.022 2685 1 I Unity : UnityEngine.CSSLayout.Native:.cctor()
01-09 15:16:42.022 2685 1 I Unity : UnityEngine.DebugLogHandler:Internal_Log(LogType, String, Object)
01-09 15:16:42.022 2685 1 I Unity : UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
01-09 15:16:42.022 2685 1 I Unity : UnityEngine.Logger:Log(LogType, Object)
01-09 15:16:42.022 2685 1 I Unity : UnityEngine.Debug:Log(Object)
01-09 15:16:42.022 2685 1 I Unity : Dissonance.Logs:GetLogLevel(LogCategory)
01-09 15:16:42.022 2685 1 I Unity : Dissonance.Logs:SendLogMessage(String, LogLevel)
01-09 15:16:42.022 2685 1 I Unity : Dissonance.Log:WriteLog(LogLevel, String)
01-09 15:16:42.022 2685 1 I Unity : Dissonance.Log:WriteLogFormat(LogLevel, String, TA)
01-09 15:16:42.022 2685 1 I Unity : Dissonance.Log:Debug(String, TA)
01-09 15:16:42.022 2685 1 I Unity : Dissonance.PlayerTrackerManager:RemoveTracker(IDissonancePlayer)
01-09 15:16:42.022 2685 1 I Unity : Dissonance.DissonanceComms:StopTracking(IDissonancePlayer)
01-09 15:16:42.022 2685 1 I Unity : Dissonance.Integrations.UNet_HLAPI.HlapiPlayer:StopTracking()
01-09 15:16:42.022 2685 1 I Unity : Dissonance.Integrations.UNet_HLAPI.HlapiPlayer:OnDisable()
01-09 15:16:42.022 2685 1 I Unity : HandyCollections.Heap.IMinHeap`1:IndexOf(Predicate`1)
01-09 15:16:42.022 2685 1 I Unity : UnityEngine.CSSLayout.Native:.cctor()
01-09 15:16:42.022 2685 1 I Unity :
01-09 15:16:42.022 2685 1 I Unity : (Filename: C:/buildslave/unity/build\Runtime/Export/Debug.bindings.h Line: 43)
01-09 15:16:42.022 2685 1 I Unity :
01-09 15:16:42.045 2685 1 I Unity : DEBUG [Dissonance:Recording] (06:16:42.031) CapturePipelineManager: Destroying
01-09 15:16:42.045 2685 1 I Unity : System.Diagnostics.StackTrace:.ctor(Int32, Boolean)
01-09 15:16:42.045 2685 1 I Unity : UnityEngine.StackTraceUtility:ExtractStackTrace()
01-09 15:16:42.045 2685 1 I Unity : UnityEngine.CSSLayout.Native:.cctor()
01-09 15:16:42.045 2685 1 I Unity : UnityEngine.DebugLogHandler:Internal_Log(LogType, String, Object)
01-09 15:16:42.045 2685 1 I Unity : UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
01-09 15:16:42.045 2685 1 I Unity : UnityEngine.Logger:Log(LogType, Object)
01-09 15:16:42.045 2685 1 I Unity : UnityEngine.Debug:Log(Object)
01-09 15:16:42.045 2685 1 I Unity : Dissonance.Logs:GetLogLevel(LogCategory)
01-09 15:16:42.045 2685 1 I Unity : Dissonance.Logs:SendLogMessage(String, LogLevel)
01-09 15:16:42.045 2685 1 I Unity : Dissonance.Log:WriteLog(LogLevel, String)
01-09 15:16:42.045 2685 1 I Unity : Dissonance.Log:Debug(String)
01-09 15:16:42.045 2685 1 I Unity : Dissonance.Audio.Capture.CapturePipelineManager:Destroy()
01-09 15:16:42.045 2685 1 I Unity : Dissonance.DissonanceComms:OnDestroy()
01-09 15:16:42.045 2685 1 I Unity : HandyCollections.Heap.IMinHeap`1:IndexOf(Predicate`1)
01-09 15:16:42.045 2685 1 I Unity : UnityEngine.CSSLayout.Native:.cctor()
01-09 15:16:42.045 2685 1 I Unity :
01-09 15:16:42.045 2685 1 I Unity : (Filename: C:/buildslave/unity/build\Runtime/Export/Debug.bindings.h Line: 43)
01-09 15:16:42.045 2685 1 I Unity :
01-09 15:16:42.050 2685 1 E Unity : NullReferenceException: A null value was found where an object instance was required.
01-09 15:16:42.050 2685 1 E Unity : at UnityEngine.ImageConversion.LoadImage (UnityEngine.Texture2D tex, System.Byte[] data) [0x00000] in <filename unknown>:0
01-09 15:16:42.050 2685 1 E Unity : at UnityEngine.JsonUtility.FromJson (System.String json, System.Type type) [0x00000] in <filename unknown>:0
01-09 15:16:42.050 2685 1 E Unity : at UnityEngine.JsonUtility.FromJson (System.String json, System.Type type) [0x00000] in <filename unknown>:0
01-09 15:16:42.050 2685 1 E Unity : at UnityEngine.JsonUtility.FromJson (System.String json, System.Type type) [0x00000] in <filename unknown>:0
01-09 15:16:42.050 2685 1 E Unity : at MagicLeap.ControllerStatusIndicator.OnDestroy () [0x00000] in <filename unknown>:0
01-09 15:16:42.050 2685 1 E Unity : at HandyCollections.Heap.IMinHeap`1[T].IndexOf (System.Predicate`1 predicate) [0x00000] in <filename unknown>:0
01-09 15:16:42.050 2685 1 E Unity : at UnityEngine.CSSLayout.Native..cctor () [0x00000] in <filename unknown>:0
01-09 15:16:42.050 2685 1 E Unity : System.Diagnostics.StackTrace:.ctor(Int32, Boolean)
01-09 15:16:42.050 2685 1 E Unity : UnityEngine.StackTraceUtility:ExtractStringFromExceptionInternal(Object, String&, String&)
01-09 15:16:42.050 2685 1 E Unity : HandyCollections.Heap.IMinHeap`1:IndexOf(Predicate`1)
01-09 15:16:42.050 2685 1 E Unity : UnityEngine.CSSLayout.Native:.cctor()
01-09 15:16:42.050 2685 1 E Unity :
01-09 15:16:42.050 2685 1 E Unity : (Filename: currently not available on il2cpp Line: -1)
01-09 15:16:42.050 2685 1 E Unity :
01-09 15:16:42.055 2685 1 E Unity : NullReferenceException: A null value was found where an object instance was required.
01-09 15:16:42.055 2685 1 E Unity : at UnityEngine.ImageConversion.LoadImage (UnityEngine.Texture2D tex, System.Byte[] data) [0x00000] in <filename unknown>:0
01-09 15:16:42.055 2685 1 E Unity : at UnityEngine.JsonUtility.FromJson (System.String json, System.Type type) [0x00000] in <filename unknown>:0
01-09 15:16:42.055 2685 1 E Unity : at UnityEngine.JsonUtility.FromJson (System.String json, System.Type type) [0x00000] in <filename unknown>:0
01-09 15:16:42.055 2685 1 E Unity : at UnityEngine.JsonUtility.FromJson (System.String json, System.Type type) [0x00000] in <filename unknown>:0
01-09 15:16:42.055 2685 1 E Unity : at MagicLeap.ControllerStatusText.OnDestroy () [0x00000] in <filename unknown>:0
01-09 15:16:42.055 2685 1 E Unity : at HandyCollections.Heap.IMinHeap`1[T].IndexOf (System.Predicate`1 predicate) [0x00000] in <filename unknown>:0
01-09 15:16:42.055 2685 1 E Unity : at UnityEngine.CSSLayout.Native..cctor () [0x00000] in <filename unknown>:0
01-09 15:16:42.055 2685 1 E Unity : System.Diagnostics.StackTrace:.ctor(Int32, Boolean)
01-09 15:16:42.055 2685 1 E Unity : UnityEngine.StackTraceUtility:ExtractStringFromExceptionInternal(Object, String&, String&)
01-09 15:16:42.055 2685 1 E Unity : HandyCollections.Heap.IMinHeap`1:IndexOf(Predicate`1)
01-09 15:16:42.055 2685 1 E Unity : UnityEngine.CSSLayout.Native:.cctor()
We have multiple devices and are very keen on assisting with looking for a solution on the Leap. We too agree that it is an issue with the codec, anything we can do to help expedite testing and implementing a solution let us know.
Thanks for your debug output Frisky!
OK I could not see the Dissonance log messages using mldb so I edited the Dissonance Log.cs script so that it would write all debug messages to a text file.
Here is the output:
[Dissonance:Core] (06:16:07.882) DissonanceComms: Loading default playback prefab:Info
DEBUG [Dissonance:Core] (06:16:14.278) PlayerTrackerManager: Got a player tracker for player 'd3d2e5c0-9dc7-4f7b-bd00-44142ed237c1' but that player doesn't exist yet:Debug
DEBUG [Dissonance:Network] (06:16:14.313) HLAPI Player Component: OnStartAuthority Name=cda581d9-7e7e-493d-a002-9f59e7500f84:Debug
DEBUG [Dissonance:Core] (06:16:14.336) PlayerTrackerManager: Associated position tracking for 'cda581d9-7e7e-493d-a002-9f59e7500f84':Debug
DEBUG [Dissonance:Core] (06:16:14.371) VoiceBroadcastTrigger: Collider entered (1):Debug
DEBUG [Dissonance:Core] (06:16:14.385) VoiceReceiptTrigger: Collider entered (1):Debug
DEBUG [Dissonance:Core] (06:16:14.408) VoiceReceiptTrigger: Recalculating token activation: 0 tokens, activated: True:Debug
DEBUG [Dissonance:Core] (06:16:14.422) Rooms: Joined chat room 'Room B':Debug
DEBUG [Dissonance:Core] (06:16:42.001) PlayerTrackerManager: Removed unlinked state tracker for 'd3d2e5c0-9dc7-4f7b-bd00-44142ed237c1' (because RemoveTracker called):Debug
DEBUG [Dissonance:Recording] (06:16:42.031) CapturePipelineManager: Destroying:Debug
Note (d3d2e5c0-9dc7-4f7b-bd00-44142ed237c1 was the ID of another device (a hololens) that was on the server) Some debug messages from that device:
DEBUG [Dissonance H:Core] (06:16:17.665) PlayerTrackerManager: Got a player tracker for player 'cda581d9-7e7e-493d-a002-9f59e7500f84' but that player doesn't exist yet
DEBUG [Dissonance H:Core] (06:16:47.303) PlayerTrackerManager: Removed unlinked state tracker for 'cda581d9-7e7e-493d-a002-9f59e7500f84' (because RemoveTracker called)
Wow, thanks for the help guys!
Unable to find an entry point named 'dissonance_opus_encoder_ctl_in' in 'opus'.
It looks like this error may be quite simple to fix, it's not saying that the plugin file (opus.so) is incompatible, just that it can't find a specific entry point dissonance_opus_encoder_ctl_in
. In Assets/Plugins/Dissonance/Core/Audio/Codecs/Opus/Opus.cs
on line 93 that method is imported in this big ugly block of preprocessor directives:
#if UNITY_ANDROID && !UNITY_EDITOR
[DllImport("opus", CallingConvention = CallingConvention.Cdecl)]
internal static extern int opus_decoder_ctl(IntPtr st, Ctl request, int value);
internal static int dissonance_opus_decoder_ctl_in(IntPtr st, Ctl request, int value)
{
return opus_decoder_ctl(st, request, value);
}
#else
#if UNITY_IOS && !UNITY_EDITOR
[DllImport("__Internal", CallingConvention = CallingConvention.Cdecl)]
#else
[DllImport("opus", CallingConvention = CallingConvention.Cdecl)]
#endif
internal static extern int dissonance_opus_decoder_ctl_in(IntPtr st, Ctl request, int value);
#endif
You can see that there is special handling for Android - we import a different method from the plugin (opus_decoder_ctl
) and wrap it in a method defined in C# (dissonance_opus_decoder_ctl_in
). If you modify this so that magic leap is handled in the same way as Android it may fix the problem.
Unfortunately I can't see any MAGIC_LEAP
directive available to the preprocessor (see docs here) so I'm not sure how best to do that (other than just defining your own directive).
Got a player tracker for player 'cda581d9-7e7e-493d-a002-9f59e7500f84' but that player doesn't exist yet Removed unlinked state tracker for 'cda581d9-7e7e-493d-a002-9f59e7500f84' (because RemoveTracker called)
This can happen if the player tracker for a player starts up before the player has joined the Dissonance session (which is quite common, the gameobject usually gets initialised before Dissonance has completed it's network handshake). The tracker gets stored in an unlinked
state and when the player eventually joins the Dissonance session it gets linked up. The second one is a bit weird (it happens after 30 seconds, definitely enough time for someone to join) but let's not worry about it for now - session initialisation is probably being disrupted by the codec initialisation exploding!
You rock @martindevans !
We checked it out and Magic Leap's directive is "PLATFORM_LUMIN".
We are about to test and report back shortly.
@martindevans Magic Leap is now appearing as a peer, and we are able to receive audio from other instances that are Magic Leap, but still not able to send audio from the leap to other peers. We've gone through all the correct permissions on the Leap's side such as using the microphone, and we have gotten the microphone to work locally. Still having trouble sending that data over the network.
@FriskyBishop How were you able to output the debug log in your very first post? We can try and output the same now with our patches in place to see where things stand. Thanks guys!
My guess is that the codec (opus.so
) is fixed but the preprocessor (libAudioPluginDissonance.so
) isn't - the preprocessor is significantly more complex so it may need to be recompiled against a magic leap specific set of libs. Definitely see if you can get a log file to confirm that.
If you go to Assets/Plugins/Dissonance/Core/Log.cs
Line 53 you'll find a single method which all Dissonance logging goes through. You can probably just replace those Debug.Log
calls with something like File.AppendAllText("log file path", log_msg);
(that's not very performant, but it should serve as a temporary hack while we're debugging).
Thanks @martindevans Let us attempt to generate the debug log and we will report back.
If it is an issue with the preprocessor needing to be recompiled, how can we assist in providing the right pieces to help achieve this goal? We are in communication now with Magic Leap as we try to resolve this issue.
If you know anything about how to recompile C++ libs I can give you the source for all the native parts of Dissonance (Just email me (martin@placeholder-software.co.uk) your invoice number). That's going to be a lot quicker than trying to get me setup with ML development environment! I'm hoping it will be a relatively simple tweak to the current android build process (especially if you have support from ML themselves), feel free to copy me in on any discussions with them regarding the build process and I'll provide whatever tips I can.
@martindevans We have an output:
01-09 15:42:17.064 4428 1 E Unity : (Filename: currently not available on il2cpp Line: -1)
01-09 15:42:17.064 4428 1 E Unity :
01-09 15:42:17.078 4428 1 E Unity : [Dissonance:Recording] (20:42:16.665) BasePreprocessingPipeline: Voice Error: Unhandled exception killed audio preprocessor thread: System.EntryPointNotFoundException: Unable to find an entry point named 'Dissonance_CreatePreprocessor' in 'AudioPluginDissonance'.
01-09 15:42:17.078 4428 1 E Unity : at UnityEngine.ImageConversion.LoadImage (UnityEngine.Texture2D tex, System.Byte[] data) [0x00000] in <filename unknown>:0
01-09 15:42:17.078 4428 1 E Unity : at UnityEngine.JsonUtility.FromJson (System.String json, System.Type type) [0x00000] in <filename unknown>:0
01-09 15:42:17.078 4428 1 E Unity : at UnityEngine.JsonUtility.FromJson (System.String json, System.Type type) [0x00000] in <filename unknown>:0
01-09 15:42:17.078 4428 1 E Unity : at Dissonance.Audio.Capture.WebRtcPreprocessingPipeline+WebRtcPreprocessor.Dissonance_CreatePreprocessor (NoiseSuppressionLevels nsLevel, AecSuppressionLevels aecLevel, Boolean aecDelayAgnostic, Boolean aecExtended, Boolean aecRefined, AecmRoutingMode aecmRoutingMode, Boolean aecmComfortNoise) [0x00000] in <filename unknown>:0
01-09 15:42:17.078 4428 1 E Unity : at Dissonance.Audio.Capture.WebRtcPreprocessingPipeline+WebRtcPreprocessor.CreatePreprocessor () [0x00000] in <filename unknown>:0
01-09 15:42:17.078 4428 1 E Unity : at Dissonance.Audio.Capture.WebRtcPreprocessingPipeline+WebRtcPreprocessor.Reset () [0x00000] in <filename unknown>:0
01-09 15:42:17.078 4428 1 E Unity : at Dissonance.Audio.Capture.WebRtcPreprocessingPipeline.ApplyReset () [0x00000] in <filename unknown>:0
01-09 15:42:17.078 4428 1 E Unity : at Dissonance.Audio.Capture.BasePreprocessingPipeline.ThreadEntry () [0x00000] in <filename unknown>:0
01-09 15:42:17.078 4428 1 E Unity : at System.Action.Invoke () [0x00000] in <filename unknown>:0
01-09 15:42:17.078 4428 1 E Unity : at System.Threading.ThreadStart.Invoke () [0x00000] in <filename unknown>:0
01-09 15:42:17.078 4428 1 E Unity : at Dissonance.IChannel`1[T].get_TargetId () [0x00000] in <filename unknown>:0
01-09 15:42:17.078 4428 1 E Unity : at UnityEngine.CSSLayout.Native..cctor () [0x00000] in <filename unknown>:0
01-09 15:42:17.078 4428 1 E Unity : at UnityEngine.CSSLayout.Native..cctor () [0x00000] in <filename unknown>:0
01-09 15:42:17.078 4428 1 E Unity : at UnityEngine.ImageConversion.LoadImage (UnityEngine.Texture2D tex, System.Byte[] data) [0x00000] in <filename unknown>:0
01-09 15:42:17.078 4428 1 E Unity : at UnityEngine.ImageConversion.LoadImage (UnityEngine.Texture2D tex, System.Byte[] data) [0x00000] in <filename unknown>:0 ! Error ID: 02EB75C0-1E12-4109-BFD2-64645C14BD5F
01-09 15:42:17.078 4428 1 E Unity : System.Diagnostics.StackTrace:.ctor(Int32, Boolean)
01-09 15:42:17.078 4428 1 E Unity : UnityEngine.StackTraceUtility:ExtractStackTrace()
01-09 15:42:17.078 4428 1 E Unity : UnityEngine.CSSLayout.Native:.cctor()
01-09 15:42:17.078 4428 1 E Unity : UnityEngine.DebugLogHandler:Internal_Log(LogType, String, Object)
01-09 15:42:17.078 4428 1 E Unity : UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
01-09 15:42:17.078 4428 1 E Unity : UnityEngine.Logger:Log(LogType, Object)
01-09 15:42:17.078 4428 1 E Unity : UnityEngine.Debug:LogError(Object)
01-09 15:42:17.078 4428 1 E Unity : Dissonance.Logs:GetLogLevel(LogCategory)
01-09 15:42:17.078 4428 1 E Unity : Dissonance.Logs:WriteMultithreadedLogs()
01-09 15:42:17.078 4428 1 E Unity : Dissonance.DissonanceComms:Update()
01-09 15:42:17.078 4428 1 E Unity : Dissonance.IChannel`1:get_TargetId()
01-09 15:42:17.078 4428 1 E Unity : UnityEngine.CSSLayout.Native:.cctor()
Well that definitely confirms my suspicions about the preprocessor being the problem! Do you want to email me and take a stab at adapting the android build process yourself?
@martindevans Shot you an email, thanks!
@FriskyBishop How were you able to output the debug log in your very first post? We can try and output the same now with our patches in place to see where things stand. Thanks guys!
@aethoco ah i see you have a log - I was using mldb log https://creator.magicleap.com/learn/guides/sdk-mldb-debugging-apps
To get the debug log messages from dissonance to save to a text file I added the following to the Log.cs script ...
at line 93
internal static void SendLogMessage(string message, LogLevel level)
{
string _l = level.ToString();
Log.logit(message, _l);
#if NCRUNCH
Console.WriteLine(message);
#else
var msg = new LogMessage(message, level);
if (_main == null || _main == Thread.CurrentThread)
msg.Log();
else
LogsFromOtherThreads.TryWrite(msg);
#endif
}
#endregion
}
then inside "public class Log" I added a simple write function
public static void logit(string _a, string _b)
{
string folderName = Application.persistentDataPath + "/theLog.txt";
if (!File.Exists(folderName))
{
File.Create(folderName).Dispose();
}
File.AppendAllText(folderName, _a + ":" + _b + Environment.NewLine);
}
this should write a file to the /documents/C2 folder in the app directory
@FriskyBishop Yup - we realized mldb with enabling the Dissonance debug was pretty straightforward after poking around a bit.
ML is looking internally to see if someone can help us compile the new preprocessor for Lumen OS. Will report back once we know more.
@martindevans Magic Leap is now appearing as a peer, and we are able to receive audio from other instances that are Magic Leap, but still not able to send audio from the leap to other peers. We've gone through all the correct permissions on the Leap's side such as using the microphone, and we have gotten the microphone to work locally. Still having trouble sending that data over the network.
Can also confirm this I changed all the 'UNITY_ANDROID' to 'PLATFORM_LUMIN' in Opus.cs.
Not sure if the logs will be helpful anymore but here we are ...
Here is the dissonance debug log:
[Dissonance:Core] (06:16:07.882) DissonanceComms: Loading default playback prefab:Info
DEBUG [Dissonance:Core] (06:16:14.278) PlayerTrackerManager: Got a player tracker for player 'd3d2e5c0-9dc7-4f7b-bd00-44142ed237c1' but that player doesn't exist yet:Debug
DEBUG [Dissonance:Network] (06:16:14.313) HLAPI Player Component: OnStartAuthority Name=cda581d9-7e7e-493d-a002-9f59e7500f84:Debug
DEBUG [Dissonance:Core] (06:16:14.336) PlayerTrackerManager: Associated position tracking for 'cda581d9-7e7e-493d-a002-9f59e7500f84':Debug
DEBUG [Dissonance:Core] (06:16:14.371) VoiceBroadcastTrigger: Collider entered (1):Debug
DEBUG [Dissonance:Core] (06:16:14.385) VoiceReceiptTrigger: Collider entered (1):Debug
DEBUG [Dissonance:Core] (06:16:14.408) VoiceReceiptTrigger: Recalculating token activation: 0 tokens, activated: True:Debug
DEBUG [Dissonance:Core] (06:16:14.422) Rooms: Joined chat room 'Room B':Debug
DEBUG [Dissonance:Core] (06:16:42.001) PlayerTrackerManager: Removed unlinked state tracker for 'd3d2e5c0-9dc7-4f7b-bd00-44142ed237c1' (because RemoveTracker called):Debug
DEBUG [Dissonance:Recording] (06:16:42.031) CapturePipelineManager: Destroying:Debug
[Dissonance:Core] (05:50:02.508) DissonanceComms: Loading default playback prefab:Info
DEBUG [Dissonance:Recording] (05:50:02.562) CapturePipelineManager: Not changing microphone device from '' to '' (equivalent devices):Debug
[Dissonance:Core] (05:50:02.579) DissonanceComms: Starting Dissonance Voice Comms
- Network: [DissonanceSetup (Dissonance.Integrations.UNet_HLAPI.HlapiCommsNetwork)]
- Quality Settings: [Quality: Low, FrameSize: Medium, FEC: True, DenoiseAmount: High, VoiceDuckLevel: 0.75]
- Codec: [Codec: Opus, FrameSize: 1920, SampleRate: 48kHz]:Info
DEBUG [Dissonance:Network] (05:50:08.857) HlapiCommsNetwork: Starting network session as Client:Debug
DEBUG [Dissonance:Recording] (05:50:08.893) CapturePipelineManager: Restarting transmission pipeline: 'Network mode changed':Debug
[Dissonance:Recording] (05:50:09.017) BasicMicrophoneCapture: Began mic capture (SampleRate:16000Hz, FrameSize:320, Buffer Limit:2^11, Latency:20ms, Device:''):Info
DEBUG [Dissonance:Recording] (05:50:09.035) BasePreprocessingPipeline: Started preprocessor thread. Internal latency: 0ms, Total stream latency:20ms:Debug
DEBUG [Dissonance:Recording] (05:50:09.145) BasicMicrophoneCapture: Trying to read 1024 samples, growing read buffer space to 2047:Debug
DEBUG [Dissonance:Core] (05:50:09.167) PlayerTrackerManager: Got a player tracker for player 'cb17aaaa-d633-4bf4-bef6-e352de9db844' but that player doesn't exist yet:Debug
DEBUG [Dissonance:Network] (05:50:09.200) HLAPI Player Component: OnStartAuthority Name=99a6faab-8120-4235-9d7b-4f8e30c314d0:Debug
DEBUG [Dissonance:Core] (05:50:09.224) PlayerTrackerManager: Associated position tracking for '99a6faab-8120-4235-9d7b-4f8e30c314d0':Debug
[Dissonance:Network] (05:50:09.255) ConnectionNegotiator`1: Received handshake response from server, joined session '556312583':Info
DEBUG [Dissonance:Network] (05:50:09.285) SlaveClientCollection`1: Sent local client state:Debug
DEBUG [Dissonance:Core] (05:50:09.318) VoiceBroadcastTrigger: Collider entered (1):Debug
DEBUG [Dissonance:Core] (05:50:09.332) VoiceReceiptTrigger: Collider entered (1):Debug
DEBUG [Dissonance:Core] (05:50:09.356) DissonanceComms: Net_PlayerJoined. playerId:cb17aaaa-d633-4bf4-bef6-e352de9db844 codecSettings:Codec: Opus, FrameSize: 1920, SampleRate: 48kHz:Debug
DEBUG [Dissonance:Core] (05:50:09.382) PlayerTrackerManager: Linked an unlinked player tracker for player 'cb17aaaa-d633-4bf4-bef6-e352de9db844':Debug
DEBUG [Dissonance:Playback] (05:50:09.409) SamplePlaybackComponent: Resetting playback component:Debug
DEBUG [Dissonance:Recording] (05:50:09.443) BasicMicrophoneCapture: Trying to read 3072 samples, growing read buffer space to 4095:Debug
DEBUG [Dissonance:Core] (05:50:09.457) VoiceReceiptTrigger: Recalculating token activation: 0 tokens, activated: True:Debug
DEBUG [Dissonance:Core] (05:50:09.471) Rooms: Joined chat room 'Room B':Debug
[Dissonance:Recording] (05:50:09.589) BasePreprocessingPipeline: Lost 320 samples in the preprocessor (buffer full), injecting silence to compensate:Warn
and his message was being spammed repeatedly in the mldb log:
01-10 14:50:18.129 1978 1 W Unity : [Dissonance:Recording] (05:50:18.109) BasePreprocessingPipeline: Lost 320 samples in the preprocessor (buffer full), injecting silence to compensate
01-10 14:50:18.129 1978 1 W Unity : System.Diagnostics.StackTrace:.ctor(Int32, Boolean)
01-10 14:50:18.129 1978 1 W Unity : UnityEngine.StackTraceUtility:ExtractStackTrace()
01-10 14:50:18.129 1978 1 W Unity : UnityEngine.CSSLayout.Native:.cctor()
01-10 14:50:18.129 1978 1 W Unity : UnityEngine.DebugLogHandler:Internal_Log(LogType, String, Object)
01-10 14:50:18.129 1978 1 W Unity : UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
01-10 14:50:18.129 1978 1 W Unity : UnityEngine.Logger:Log(LogType, Object)
01-10 14:50:18.129 1978 1 W Unity : UnityEngine.Debug:LogWarning(Object)
01-10 14:50:18.129 1978 1 W Unity : Dissonance.Logs:GetLogLevel(LogCategory)
01-10 14:50:18.129 1978 1 W Unity : Dissonance.Logs:SendLogMessage(String, LogLevel)
01-10 14:50:18.129 1978 1 W Unity : Dissonance.Log:WriteLog(LogLevel, String)
01-10 14:50:18.129 1978 1 W Unity : Dissonance.Log:WriteLogFormat(LogLevel, String, TA)
01-10 14:50:18.129 1978 1 W Unity : Dissonance.Log:Warn(String, TA)
01-10 14:50:18.129 1978 1 W Unity : Dissonance.Audio.Capture.BasePreprocessingPipeline:Dissonance.Audio.Capture.IMicrophoneSubscriber.ReceiveMicrophoneData(ArraySegment`1, WaveFormat)
01-10 14:50:18.129 1978 1 W Unity : Dissonance.Audio.Capture.BasicMicrophoneCapture:SendFrame()
01-10 14:50:18.129 1978 1 W Unity : Dissonance.Audio.Capture.BasicMicrophoneCapture:ConsumeSamples(ArraySegment`1)
01-10 14:50:18.129 1978 1 W Unity : Dissonance.Audio.Capture.BasicMicrophoneCapture:DrainMicSamples()
01-10 14:50:18.129 1978 1 W Unity : Dissonance.Audio.Capture.BasicMicrophoneCapture:UpdateSubscribers()
01-10 14:50:18.129 1978 1 W Unity : Dissonance.Audio.Capture.CapturePipelineManager:Update(Boolean, Single)
01-10 14:50:18.129 1978 1 W Unity : Dissonance.DissonanceComms:Update()
01-10 14:50:18.129 1978 1 W Unity : System.Collections.Generic.ICollection`1:get_IsReadOnly()
01-10 14:50:18.129 1978 1 W Unity : UnityEngine.CSSLayout.Native:.cctor()
01-10 14:50:18.129 1978 1 W Unity :
01-10 14:50:18.129 1978 1 W Unity : (Filename: C:/buildslave/unity/build\Runtime/Export/Debug.bindings.h Line: 43)
Thanks @FriskyBishop I've been asking around, and it seems the best answer so far is that ML rolled their own: "The people I know that have done this so far all simply "rolled their own" solutions connecting our MLAudio API to a custom compression and network transport layer."
That said, if we can solve this issue, it would make Dissonance the de facto way to implement VOIP into their ML projects. Will keep you guys updated as we continue to hunt for assistance.
@FriskyBishop That log is surprising, as far as I can see you never have the Unhandled exception killed audio preprocessor thread: System.EntryPointNotFoundException: Unable to find an entry point named 'Dissonance_CreatePreprocessor' in 'AudioPluginDissonance'.
message?
@aethoco I've sent you the source code by email. I'll try to make some time next week to investigate into building it for ML.
connecting our MLAudio API to a custom compression and network transport layer
I know I may be slightly biased but it's really not that simple! Just stuffing audio into a network stream handles none of the cases required for good audio quality with bad network conditions >_<
A dev at ML has been reviewing this thread and suggested the following:
"I'm looking at the error log posted on the GitHub issue and this may be a different problem.. The error log says it's trying to consume a 16khz stream, but our World CAmera stream only support 48khz.
So everything may be working, but you may be getting silence from the mics. (Looking at MLAudioCreateInputFromWorldCapture in the API)"
We do have this System.EntryPointNotFoundException
which will definitely kill the audio encoding system. I don't think that can be caused by an incorrect sample rate. @FriskyBishop doesn't seem to have that which is odd - did either of you change anything else (e.g. plugin import settings)?
The 16kHz sample rate being referred to is presumably from this log message:
[Dissonance:Recording] (05:50:09.017) BasicMicrophoneCapture: Began mic capture (SampleRate:16000Hz, FrameSize:320, Buffer Limit:2^11, Latency:20ms, Device:''):Info
The sample rate is chosen by this code (Assets/Plugins/Dissonance/Core/Audio/Capture/BasicMicrophoneCapture.cs Line 64
):
Microphone.GetDeviceCaps(_micName, out minFreq, out maxFreq);
var sampleRate = minFreq == 0 && maxFreq == 0 ? 48000 : Mathf.Clamp(48000, minFreq, maxFreq);
GetDeviceCaps returns min:0 max:0
if the device can handle any rate, in which case we default to 48kHz. Otherwise we select the closest value to 48kHz within the valid range.
If we're selecting 16,000 then GetDeviceCaps
has returned the wrong limits. Could you check what values this returns:
int minFreq, maxFreq;
Microphone.GetDeviceCaps("", out minFreq, out maxFreq);
If the ML only support 48kHz then both these values should be 48,000
. If they're not we can report a bug to someone.
As a workaround you could just hardcode 48kHz as the sample rate here.
#if PLATFORM_LUMIN
var sampleRate = 48000;
#else
Microphone.GetDeviceCaps(_micName, out minFreq, out maxFreq);
var sampleRate = minFreq == 0 && maxFreq == 0 ? 48000 : Mathf.Clamp(48000, minFreq, maxFreq);
#endif
@martindevans Thanks Martin. We will attempt the 48khz hardcode patch this afternoon and let everyone know.
@martindevans We attempted the 48khz hardcode patch and still at the same result, our debug logs:
First message proves we are capturing the mic at 48000:
01-11 09:01:18.045 1585 1 I Unity : [Dissonance:Recording] (14:01:18.026) BasicMicrophoneCapture: Began mic capture (SampleRate:48000Hz, FrameSize:960, Buffer Limit:2^13, Latency:20ms, Device:'') 01-11 09:01:18.045 1585 1 I Unity : System.Diagnostics.StackTrace:.ctor(Int32, Boolean) 01-11 09:01:18.045 1585 1 I Unity : UnityEngine.StackTraceUtility:ExtractStackTrace() 01-11 09:01:18.045 1585 1 I Unity : UnityEngine.CSSLayout.Native:.cctor() 01-11 09:01:18.045 1585 1 I Unity : UnityEngine.DebugLogHandler:Internal_Log(LogType, String, Object) 01-11 09:01:18.045 1585 1 I Unity : UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[]) 01-11 09:01:18.045 1585 1 I Unity : UnityEngine.Logger:Log(LogType, Object) 01-11 09:01:18.045 1585 1 I Unity : UnityEngine.Debug:Log(Object) 01-11 09:01:18.045 1585 1 I Unity : Dissonance.Logs:GetLogLevel(LogCategory) 01-11 09:01:18.045 1585 1 I Unity : Dissonance.Logs:SendLogMessage(String, LogLevel) 01-11 09:01:18.045 1585 1 I Unity : Dissonance.Log:WriteLog(LogLevel, String) 01-11 09:01:18.045 1585 1 I Unity : Dissonance.Log:WriteLogFormat(LogLevel, String, TA, TB, TC, TD, TE) 01-11 09:01:18.045 1585 1 I Unity : Dissonance.Log:Info(String, TA, TB, TC, TD, TE) 01-11 09:01:18.045 1585 1 I Unity : Dissonance.Audio.Capture.BasicMicrophoneCapture:StartCapture(String) 01-11 09:01:18.045 1585 1 I Unity : Dissonance.Audio.Capture.CapturePipelineManager:RestartTransmissionPipeline(String) 01-11 09:01:18.045 1585 1 I Unity : Dissonance.Audio.Capture.CapturePipelineManager:Update(Boolean, Single) 01-11 09:01:18.045 1585 1 I Unity : Dissonance.DissonanceComms:Update() 01-11 09:01:18.045 1585 1 I Unity : UnityEngine.ImageConversion:LoadImage(Texture2D, Byte[]) 01-11 09:01:18.045 1585 1 I Unity : UnityEngine.CSSLayout.Native:.cctor()
Second message proves we are stuck at the preprocessor:
01-11 09:01:18.337 1585 1 E Unity : [Dissonance:Recording] (14:01:18.051) BasePreprocessingPipeline: Voice Error: Unhandled exception killed audio preprocessor thread: System.EntryPointNotFoundException: Unable to find an entry point named 'Dissonance_CreatePreprocessor' in 'AudioPluginDissonance'.
Third message we are being spammed with at every frame - Is this message being sent due to the preprocessor error?
01-11 09:01:18.972 1585 1 W Unity : [Dissonance:Recording] (14:01:18.951) BasePreprocessingPipeline: Lost 960 samples in the preprocessor (buffer full), injecting silence to compensate
Thanks!
Third message we are being spammed with at every frame
That's caused by the earlier exception. We have the main Unity thread putting raw mic audio into the input buffer and the background preprocessor thread draining the buffer... but the background thread is dead so the input buffer is permanently full and so audio is being discarded.
BasePreprocessingPipeline: Voice Error: Unhandled exception killed audio preprocessor thread
I'm really interested why @FriskyBishop doesn't seem to get this exception. Could you (@FriskyBishop) go to Assets/Plugins/Dissonance/Core/Audio/Capture/WebRtcPreprocessingPipeline.cs Line 310
and put in some log messages:
Debug.Log("--> About to create preprocessor");
var preprocessor = Dissonance_CreatePreprocessor(
NoiseSuppressionLevel,
pcLevel,
instance.AecDelayAgnostic,
instance.AecExtendedFilter,
instance.AecRefinedAdaptiveFilter,
mobLevel,
instance.AecmComfortNoise
);
Debug.Log(string.Format("--> Created Preprocessor: {0}", preprocessor));
return preprocessor
If that prints both messages (when run on the Magic Leap) then we know the AudioPluginDissonance.so
plugin is loaded and the Dissonance_CreatePreprocessor
entry point is being found.
Did either of your change the import settings for the files found in https://github.com/TomGillen/Dissonance/tree/master/Assets/Plugins/Dissonance/Plugins/Android/libs/armeabi-v7a
?
@martindevans We did a complete install of Dissonance without doing anything other than prompted by the Getting Started Guide.
However this might be it on our end - we're building specifically for Lumin.
https://i.imgur.com/sCOa44O.jpg
Lumen was unticked
That definitely should be ticked. I must admit I'm a little confused how opus is working if that isn't checked on opus.so
😕
Ticking Lumen, the app builds fine - with no errors - but I can't deploy the app on the device - I click the app and it just kicks me back to the main menu every time.
To add: The ML debug log output nothing
Can you ask your magic leap contact what they think of that? It seems odd that the program dies with nothing at all written to the log. If possible I'd like to get confirmation that's the behaviour they'd expect for an incompatible binary plugin.
My plan at the moment is to download the Lumin SDK on Monday and see if I can adapt the android build process for AudioPluginDissonance.so
to simply use the Lumin SDK instead of the android SDK.
@martindevans Will do.
Good luck for Monday, let us know if we can help test as you move through that process.
I've managed to get the plugin to build with the mabu build tool. This initially threw some Magic Leap specific errors which reassures me that it was actually building for the right thing! I've emailed the build to @aethoco so we'll see if this actually works on the device!
@martindevans like @aethoco I did not check Lumen on any of the plugins I tried checking the one you suggested, it builds and installs on the device but will not run.
I will add the debug code and test today
Could you email the build to me too? I'd like to test it matt@thedesignium.com
@FriskyBishop I've forwarded you the email I sent to @aethoco earlier today with my test build attached. Make sure to hardcode the 48kHz change as mentioned in this post when you're testing it :)
I added the debug to the WebRtcPreprocessingPipeline.cs
Debug.Log("--> About to create preprocessor");
var preprocessor = Dissonance_CreatePreprocessor(
NoiseSuppressionLevel,
pcLevel,
instance.AecDelayAgnostic,
instance.AecExtendedFilter,
instance.AecRefinedAdaptiveFilter,
mobLevel,
instance.AecmComfortNoise
);
Debug.Log(string.Format("--> Created Preprocessor: {0}", preprocessor));
return preprocessor
and tested the new plugin ...
Same results as before - Magic Leap can receive but is not sending data
I edited the basic mic script
I added
#if PLATFORM_LUMIN
var sampleRate = 48000;
#else
Microphone.GetDeviceCaps(_micName, out minFreq, out maxFreq);
var sampleRate = minFreq == 0 && maxFreq == 0 ? 48000 : Mathf.Clamp(48000, minFreq, maxFreq);
#endif
at line 64 of BasicMicrophoneCapture.cs
but maybe it didnt work or I edited it wrong as I get alot of debug errors from BasePreprocessingPipeline and BasicMicrophoneCapture
On start-up
01-15 10:06:28.408 2087 1 I Unity : [Dissonance:Core] (01:06:28.394) DissonanceComms: Loading default playback prefab
01-15 10:06:28.408 2087 1 I Unity : System.Diagnostics.StackTrace:.ctor(Int32, Boolean)
01-15 10:06:28.408 2087 1 I Unity : UnityEngine.StackTraceUtility:ExtractStackTrace()
01-15 10:06:28.408 2087 1 I Unity : UnityEngine.CSSLayout.Native:.cctor()
01-15 10:06:28.408 2087 1 I Unity : UnityEngine.DebugLogHandler:Internal_Log(LogType, String, Object)
01-15 10:06:28.408 2087 1 I Unity : UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
01-15 10:06:28.408 2087 1 I Unity : UnityEngine.Logger:Log(LogType, Object)
01-15 10:06:28.408 2087 1 I Unity : UnityEngine.Debug:Log(Object)
01-15 10:06:28.408 2087 1 I Unity : Dissonance.Logs:GetLogLevel(LogCategory)
01-15 10:06:28.408 2087 1 I Unity : Dissonance.Logs:SendLogMessage(String, LogLevel)
01-15 10:06:28.408 2087 1 I Unity : Dissonance.Log:WriteLog(LogLevel, String)
01-15 10:06:28.408 2087 1 I Unity : Dissonance.Log:Info(String)
01-15 10:06:28.408 2087 1 I Unity : Dissonance.DissonanceComms:Start()
01-15 10:06:28.408 2087 1 I Unity : UnityEngine.ImageConversion:LoadImage(Texture2D, Byte[])
01-15 10:06:28.408 2087 1 I Unity : UnityEngine.CSSLayout.Native:.cctor()
01-15 10:06:28.408 2087 1 I Unity :
01-15 10:06:28.408 2087 1 I Unity : (Filename: C:/buildslave/unity/build\Runtime/Export/Debug.bindings.h Line: 43)
01-15 10:06:28.408 2087 1 I Unity :
01-15 10:06:28.415 2087 1 D Unity : Looking for Plugin opus -> /package/bin
01-15 10:06:28.415 2087 1 D Unity : Looking for Plugin opus -> /system/lib64
01-15 10:06:28.464 2087 1 I Unity : DEBUG [Dissonance:Recording] (01:06:28.450) CapturePipelineManager: Not changing microphone device from '' to '' (equivalent devices)
01-15 10:06:28.464 2087 1 I Unity : System.Diagnostics.StackTrace:.ctor(Int32, Boolean)
01-15 10:06:28.464 2087 1 I Unity : UnityEngine.StackTraceUtility:ExtractStackTrace()
01-15 10:06:28.464 2087 1 I Unity : UnityEngine.CSSLayout.Native:.cctor()
01-15 10:06:28.464 2087 1 I Unity : UnityEngine.DebugLogHandler:Internal_Log(LogType, String, Object)
01-15 10:06:28.464 2087 1 I Unity : UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
01-15 10:06:28.464 2087 1 I Unity : UnityEngine.Logger:Log(LogType, Object)
01-15 10:06:28.464 2087 1 I Unity : UnityEngine.Debug:Log(Object)
01-15 10:06:28.464 2087 1 I Unity : Dissonance.Logs:GetLogLevel(LogCategory)
01-15 10:06:28.464 2087 1 I Unity : Dissonance.Logs:SendLogMessage(String, LogLevel)
01-15 10:06:28.464 2087 1 I Unity : Dissonance.Log:WriteLog(LogLevel, String)
01-15 10:06:28.464 2087 1 I Unity : Dissonance.Log:WriteLogFormat(LogLevel, String, TA, TB)
01-15 10:06:28.464 2087 1 I Unity : Dissonance.Log:Debug(String, TA, TB)
01-15 10:06:28.464 2087 1 I Unity : Dissonance.Audio.Capture.CapturePipelineManager:set_MicrophoneName(String)
01-15 10:06:28.464 2087 1 I Unity : Dissonance.DissonanceComms:Start()
01-15 10:06:28.464 2087 1 I Unity : UnityEngine.ImageConversion:LoadImage(Texture2D, Byte[])
01-15 10:06:28.464 2087 1 I Unity : UnityEngine.CSSLayout.Native:.cctor()
01-15 10:06:28.464 2087 1 I Unity :
01-15 10:06:28.464 2087 1 I Unity : (Filename: C:/buildslave/unity/build\Runtime/Export/Debug.bindings.h Line: 43)
01-15 10:06:28.464 2087 1 I Unity :
01-15 10:06:28.480 2087 1 I Unity : [Dissonance:Core] (01:06:28.467) DissonanceComms: Starting Dissonance Voice Comms
01-15 10:06:28.480 2087 1 I Unity : - Network: [DissonanceSetup (Dissonance.Integrations.UNet_HLAPI.HlapiCommsNetwork)]
01-15 10:06:28.480 2087 1 I Unity : - Quality Settings: [Quality: Low, FrameSize: Medium, FEC: True, DenoiseAmount: High, VoiceDuckLevel: 0.75]
01-15 10:06:28.480 2087 1 I Unity : - Codec: [Codec: Opus, FrameSize: 1920, SampleRate: 48kHz]
01-15 10:06:28.480 2087 1 I Unity : System.Diagnostics.StackTrace:.ctor(Int32, Boolean)
01-15 10:06:28.480 2087 1 I Unity : UnityEngine.StackTraceUtility:ExtractStackTrace()
01-15 10:06:28.480 2087 1 I Unity : UnityEngine.CSSLayout.Native:.cctor()
01-15 10:06:28.480 2087 1 I Unity : UnityEngine.DebugLogHandler:Internal_Log(LogType, String, Object)
01-15 10:06:28.480 2087 1 I Unity : UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
01-15 10:06:28.480 2087 1 I Unity : UnityEngine.Logger:Log(LogType, Object)
01-15 10:06:28.480 2087 1 I Unity : UnityEngine.Debug:Log(Object)
01-15 10:06:28.480 2087 1 I Unity : Dissonance.Logs:GetLogLevel(LogCategory)
01-15 10:06:28.480 2087 1 I Unity : Dissonance.Logs:SendLogMessage(String, LogLevel)
01-15 10:06:28.480 2087 1 I Unity : Dissonance.Log:WriteLog(LogLevel, String)
01-15 10:06:28.480 2087 1 I Unity : Dissonance.Log:WriteLogFormat(LogLevel, String, TA, TB, TC)
01-15 10:06:28.480 2087 1 I Unity : Dissonance.Log:Info(String, TA, TB, TC)
01-15 10:06:28.480 2087 1 I Unity : Dissonance.DissonanceComms:Start()
01-15 10:06:28.480 2087 1 I Unity : UnityEngine.ImageConversion:LoadImage(Texture2D, Byte[])
01-15 10:06:28.480 2087 1 I Unity : UnityEngine.CSSLayout.Native:.cctor()
01-15 10:06:28.480 2087 1 I Unity :
01-15 10:06:28.480 2087 1 I Unity : (Filename: C:/buildslave/unity/build\Runtime/Export/Debug.bindings.h Line: 43)
01-15 10:06:28.480 2087 1 I Unity :
01-15 10:06:28.486 2087 1 D Unity : Looking for Plugin ml_platform -> /package/bin
01-15 10:06:28.486 2087 1 D Unity : Looking for Plugin ml_platform -> /system/lib64
01-15 10:06:28.498 2087 1 E Unity : Error: MagicLeapDevice.Update failed to get valid snapshot. Snapshot is NULL
01-15 10:06:28.498 2087 1 E Unity : System.Diagnostics.StackTrace:.ctor(Int32, Boolean)
01-15 10:06:28.498 2087 1 E Unity : UnityEngine.StackTraceUtility:ExtractStackTrace()
01-15 10:06:28.498 2087 1 E Unity : UnityEngine.CSSLayout.Native:.cctor()
01-15 10:06:28.498 2087 1 E Unity : UnityEngine.DebugLogHandler:Internal_Log(LogType, String, Object)
01-15 10:06:28.498 2087 1 E Unity : UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
01-15 10:06:28.498 2087 1 E Unity : UnityEngine.Logger:Log(LogType, Object)
01-15 10:06:28.498 2087 1 E Unity : UnityEngine.Debug:LogError(Object)
01-15 10:06:28.498 2087 1 E Unity : UnityEngine.XR.MagicLeap.MLPluginLog:Error(Object)
01-15 10:06:28.498 2087 1 E Unity : UnityEngine.XR.MagicLeap.MagicLeapDevice:Update()
01-15 10:06:28.498 2087 1 E Unity : UnityEngine.ImageConversion:LoadImage(Texture2D, Byte[])
01-15 10:06:28.498 2087 1 E Unity : UnityEngine.CSSLayout.Native:.cctor()
then is spams the console with this:
01-15 10:09:40.961 2087 1 I Unity : System.Diagnostics.StackTrace:.ctor(Int32, Boolean)
01-15 10:09:40.961 2087 1 I Unity : UnityEngine.StackTraceUtility:ExtractStackTrace()
01-15 10:09:40.961 2087 1 I Unity : UnityEngine.CSSLayout.Native:.cctor()
01-15 10:09:40.961 2087 1 I Unity : UnityEngine.DebugLogHandler:Internal_Log(LogType, String, Object)
01-15 10:09:40.961 2087 1 I Unity : UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
01-15 10:09:40.961 2087 1 I Unity : UnityEngine.Logger:Log(LogType, Object)
01-15 10:09:40.961 2087 1 I Unity : UnityEngine.Debug:Log(Object)
01-15 10:09:40.961 2087 1 I Unity : Dissonance.Logs:GetLogLevel(LogCategory)
01-15 10:09:40.961 2087 1 I Unity : Dissonance.Logs:SendLogMessage(String, LogLevel)
01-15 10:09:40.961 2087 1 I Unity : Dissonance.Log:WriteLog(LogLevel, String)
01-15 10:09:40.961 2087 1 I Unity : Dissonance.Log:Debug(String)
01-15 10:09:40.961 2087 1 I Unity : Dissonance.Audio.Capture.CapturePipelineManager:Destroy()
01-15 10:09:40.961 2087 1 I Unity : Dissonance.DissonanceComms:OnDestroy()
01-15 10:09:40.961 2087 1 I Unity : UnityEngine.ImageConversion:LoadImage(Texture2D, Byte[])
01-15 10:09:40.961 2087 1 I Unity : UnityEngine.CSSLayout.Native:.cctor()
01-15 10:09:40.961 2087 1 I Unity :
01-15 10:09:40.961 2087 1 I Unity : (Filename: C:/buildslave/unity/build\Runtime/Export/Debug.bindings.h Line: 43)
01-15 10:09:40.961 2087 1 I Unity :
Logs from Dissonance Debug:
[Dissonance:Core] (01:06:28.394) DissonanceComms: Loading default playback prefab:Info
DEBUG [Dissonance:Recording] (01:06:28.450) CapturePipelineManager: Not changing microphone device from '' to '' (equivalent devices):Debug
[Dissonance:Core] (01:06:28.467) DissonanceComms: Starting Dissonance Voice Comms
- Network: [DissonanceSetup (Dissonance.Integrations.UNet_HLAPI.HlapiCommsNetwork)]
- Quality Settings: [Quality: Low, FrameSize: Medium, FEC: True, DenoiseAmount: High, VoiceDuckLevel: 0.75]
- Codec: [Codec: Opus, FrameSize: 1920, SampleRate: 48kHz]:Info
DEBUG [Dissonance:Network] (01:06:34.758) HlapiCommsNetwork: Starting network session as Client:Debug
DEBUG [Dissonance:Recording] (01:06:34.793) CapturePipelineManager: Restarting transmission pipeline: 'Network mode changed':Debug
[Dissonance:Recording] (01:06:34.987) BasicMicrophoneCapture: Began mic capture (SampleRate:48000Hz, FrameSize:960, Buffer Limit:2^13, Latency:20ms, Device:''):Info
DEBUG [Dissonance:Recording] (01:06:35.005) BasePreprocessingPipeline: Started preprocessor thread. Internal latency: 0ms, Total stream latency:20ms:Debug
DEBUG [Dissonance:Core] (01:06:35.119) PlayerTrackerManager: Got a player tracker for player '8a65b737-e920-4701-b17d-d7c9656b4131' but that player doesn't exist yet:Debug
DEBUG [Dissonance:Network] (01:06:35.154) HLAPI Player Component: OnStartAuthority Name=370a9ec1-3321-42e8-94b8-20bb872636c6:Debug
DEBUG [Dissonance:Core] (01:06:35.181) PlayerTrackerManager: Associated position tracking for '370a9ec1-3321-42e8-94b8-20bb872636c6':Debug
[Dissonance:Network] (01:06:35.215) ConnectionNegotiator`1: Received handshake response from server, joined session '912044613':Info
DEBUG [Dissonance:Network] (01:06:35.248) SlaveClientCollection`1: Sent local client state:Debug
DEBUG [Dissonance:Core] (01:06:35.285) VoiceBroadcastTrigger: Collider entered (1):Debug
DEBUG [Dissonance:Core] (01:06:35.300) VoiceReceiptTrigger: Collider entered (1):Debug
DEBUG [Dissonance:Core] (01:06:35.326) DissonanceComms: Net_PlayerJoined. playerId:8a65b737-e920-4701-b17d-d7c9656b4131 codecSettings:Codec: Opus, FrameSize: 1920, SampleRate: 48kHz:Debug
DEBUG [Dissonance:Core] (01:06:35.357) PlayerTrackerManager: Linked an unlinked player tracker for player '8a65b737-e920-4701-b17d-d7c9656b4131':Debug
DEBUG [Dissonance:Playback] (01:06:35.390) SamplePlaybackComponent: Resetting playback component:Debug
DEBUG [Dissonance:Recording] (01:06:35.432) BasicMicrophoneCapture: Trying to read 14880 samples, growing read buffer space to 2047:Debug
DEBUG [Dissonance:Recording] (01:06:35.450) BasicMicrophoneCapture: Trying to read 14880 samples, growing read buffer space to 4095:Debug
DEBUG [Dissonance:Recording] (01:06:35.467) BasicMicrophoneCapture: Trying to read 14880 samples, growing read buffer space to 8191:Debug
DEBUG [Dissonance:Recording] (01:06:35.484) BasicMicrophoneCapture: Trying to read 14880 samples, growing read buffer space to 16383:Debug
[Dissonance:Recording] (01:06:35.501) BasePreprocessingPipeline: Lost 960 samples in the preprocessor (buffer full), injecting silence to compensate:Warn
[Dissonance:Recording] (01:06:35.522) BasePreprocessingPipeline: Lost 960 samples in the preprocessor (buffer full), injecting silence to compensate:Warn
[Dissonance:Recording] (01:06:35.545) BasePreprocessingPipeline: Lost 960 samples in the preprocessor (buffer full), injecting silence to compensate:Warn
[Dissonance:Recording] (01:06:35.566) BasePreprocessingPipeline: Lost 960 samples in the preprocessor (buffer full), injecting silence to compensate:Warn
[Dissonance:Recording] (01:06:35.587) BasePreprocessingPipeline: Lost 960 samples in the preprocessor (buffer full), injecting silence to compensate:Warn
[Dissonance:Recording] (01:06:35.608) BasePreprocessingPipeline: Lost 960 samples in the preprocessor (buffer full), injecting silence to compensate:Warn
[Dissonance:Recording] (01:06:35.629) BasePreprocessingPipeline: Lost 960 samples in the preprocessor (buffer full), injecting silence to compensate:Warn
DEBUG [Dissonance:Core] (01:06:35.652) VoiceReceiptTrigger: Recalculating token activation: 0 tokens, activated: True:Debug
DEBUG [Dissonance:Core] (01:06:35.668) Rooms: Joined chat room 'Room B':Debug
then spams:
[Dissonance:Recording] (01:06:36.310) BasePreprocessingPipeline: Lost 960 samples in the preprocessor (buffer full), injecting silence to compensate:Warn
Then I get:
DEBUG [Dissonance:Playback] (01:06:36.379) SpeechSessionStream: Created new speech session with buffer time of 100ms:Debug
[Dissonance:Recording] (01:06:36.413) BasicMicrophoneCapture: Insufficient buffer space, requested 18240, clamped to 16383 (dropping samples):Warn
[Dissonance:Recording] (01:06:37.269) BasicMicrophoneCapture: Insufficient buffer space, requested 28674, clamped to 16383 (dropping samples):Warn
@FriskyBishop Those log snippets don't show the --> About to create preprocessor
or --> Created Preprocessor:
messages. Were those messages in the full log and just didn't get included in these snippets, or are those events genuinely missing? The message being spammed to console is very odd, the root of that stacktrace is Dissonance.DissonanceComms:OnDestroy()
which implies Unity is (repeatedly) destroying the component?! Can you attach another component to the same gameObject which logs Unity lifecycle events, so see if Unity is doing something odd.
@aethoco Do you see similar results?
@martindevans We should have time to test later this evening, will report back then.
@martindevans Didn't work, confirmed initialized mic with correct sample rate - just still getting spammed with these messages whenever talking:
01-15 15:47:06.275 4385 1 W Unity : [Dissonance:Recording] (20:47:06.260) BasicMicrophoneCapture: Insufficient buffer space, requested 111048, clamped to 16383 (dropping samples)
01-15 15:47:06.294 4385 1 W Unity : [Dissonance:Recording] (20:47:06.275) BasePreprocessingPipeline: Lost 480 samples in the preprocessor (buffer full), injecting silence to compensate
Are there any errors like Unhandled exception killed audio preprocessor thread: <details of error>
? If not, could you check a few other things:
Assets/Plugins/Dissonance/Core/Audio/Capture/WebRtcPreprocessingPipeline.cs Line 248 public bool Process...
Insert a log message in here on entry and exit (print inputSampleRate
, input.length
and output.Length
just to double check). This should get printed for every single frame of audio to be preprocessed. this will confirm that the preprocessor is truly running and processing audio as expected. If you also log the exact timestamp we can check that the expected number of samples per second are being processed.
Assets/Plugins/Dissonance/Core/Audio/Capture/BasicMicrophoneCapture.cs Line 328 //Send frame to subscribers
Insert a log message here showing how many samples are being sent (segment.Count
) and to how many subscribers (_subscribers.Count
). This will be printed for every frame out audio produced by the mic and will confirm that the expected data is being recorded. Again if you log the timestamp we can verify that the correct number of samples are being produced.
Enable recording diagnostics (Window > Dissonance > Diagnostic Setting) for all the capture parts of the pipeline. This will record several files to disk with raw audio captured from various stages of the pipeline. The file location will be something like Dissonance_Diagnostics/MicrophoneRawAudio_<TimeStamp>
. If you need to change that search for all uses of AudioFileWriter
and substitute in whatever path you want.
Are there any errors like Unhandled exception killed audio preprocessor thread:
? If not, could you check a few other things:
There were no errors of this type. We will try out the rest of the requests tomorrow and circle back. Thanks!
Update, we have not gotten time to sort through those requests yet. @FriskyBishop have you had any luck so far?
Update, we have not gotten time to sort through those requests yet. @FriskyBishop have you had any luck so far?
@aethoco unfortunately I have not had time yet either - hopefully I can give it another try next week
Unfortunately we have not had time to check this out in detail, as we have pressing projects that have to go out the door. Hoping @FriskyBishop might have some time, as he's been smarter than us at debugging things so far. :)
Hopefully in a week or two we might have a day or two to spend debugging. @martindevans If I were in the UK I'd loan you a Leap if I could at this point!
Our magic leap has been out of office with another employee for a week or so >.< so I have not had chance to test anymore - however - news from the Magic Leap forums
Magic Leap are sending a device my way, hopefully that will arrive within a week or two and I can do all the testing myself. They're also going to get an engineer to work with me next week to help with porting Dissonance.
Great to hear @martindevans ! Glad to see the massive effort ML is going through to help resolve this, big thanks to you and their team for the push. Also, enjoy playing with the MLO, it's a real eye opener :)
Dissonance 6.3.1 just went live on the asset store, check out the release notes for all the details. This version includes magic leap support! Thankyou everyone here for your help with testing this. Please consider giving us a review on the asset store if you're happy with how this went :)
Hoping to get support for Dissonance to work on the Magic Leap.
Currently, the Magic Leap can join a Unet server, and seems to connect to dissonance - the dissonanceComms script gets a unique 'Local Player ID' that is consistent across all other connected instances of unity connected to the server.
However, the Magic Leap does not appear as a 'peer' on any other instance of unity and cannot send or receive voice.
I don't seem to be getting any errors and the same setup is working on other android devices.