[feature] Magic Leap Support

AAvrilML commented 5 years ago

Hoping to get support for Dissonance to work on the Magic Leap.

Currently, the Magic Leap can join a Unet server, and seems to connect to dissonance - the dissonanceComms script gets a unique 'Local Player ID' that is consistent across all other connected instances of unity connected to the server.

However, the Magic Leap does not appear as a 'peer' on any other instance of unity and cannot send or receive voice.

I don't seem to be getting any errors and the same setup is working on other android devices.

martindevans commented 5 years ago

Magic Leap isn't a platform we properly support at the moment, although it uses Android I think it's a heavily modified version which won't load our binary dependencies (audio preprocessor and opus codec). However, I would expect that to throw some kind of error if it was the problem. Could you turn up the logging settings for Dissonance (Window > Dissonance > Diagnostic Settings) to Debug for all categories and see me a log (paste it here or email to martin@placeholder-software.co.uk). Thanks :)

aethoco commented 5 years ago

Hi @martindevans has there been any updates or plans to support Magic Leap? Thanks!

martindevans commented 5 years ago

We don't have any concrete plans for magic leap support at the moment because we don't have the actual device to test on and it's not been very highly requested. As it stands we'd need someone who is willing to work with us testing on the actual device.

mechpil0t commented 5 years ago

@martindevans i just tried my dissonance project on the magic leap here a bit of a log from booting the start up of dissonance running on magic leap

Unity version: Unity 2018.1.6f1-MLTP7 (64-bit) Dissonance: HLAPI v6.2.6 (2018-10-31)

`01-09 14:08:56.188  2142     1 D Unity   : Looking for Plugin opus -> /package/bin
01-09 14:08:56.188  2142     1 D Unity   : Looking for Plugin opus -> /system/lib64
01-09 14:08:56.194  2142     1 E Unity   : EntryPointNotFoundException: Unable to find an entry point named 'dissonance_opus_encoder_ctl_in' in 'opus'.
01-09 14:08:56.194  2142     1 E Unity   :   at UnityEngine.JsonUtility.FromJson (System.String json, System.Type type) [0x00000] in <filename unknown>:0
01-09 14:08:56.194  2142     1 E Unity   :   at UnityEngine.JsonUtility.FromJson (System.String json, System.Type type) [0x00000] in <filename unknown>:0
01-09 14:08:56.194  2142     1 E Unity   :   at Dissonance.Audio.Codecs.Opus.OpusNative+OpusNativeMethods.dissonance_opus_encoder_ctl_in (IntPtr st, Ctl request, Int32 value) [0x00000] in <filename unknown>:0
01-09 14:08:56.194  2142     1 E Unity   :   at Dissonance.Audio.Codecs.Opus.OpusNative+OpusEncoder.OpusCtlIn (Ctl ctl, Int32 value) [0x00000] in <filename unknown>:0
01-09 14:08:56.194  2142     1 E Unity   :   at Dissonance.Audio.Codecs.Opus.OpusNative+OpusEncoder.set_EnableForwardErrorCorrection (Boolean value) [0x00000] in <filename unknown>:0
01-09 14:08:56.194  2142     1 E Unity   :   at Dissonance.Audio.Codecs.Opus.OpusEncoder..ctor (AudioQuality quality, FrameSize frameSize, Boolean fec) [0x00000] in <filename unknown>:0
01-09 14:08:56.194  2142     1 E Unity   :   at Dissonance.CodecSettingsLoader.CreateEncoder (AudioQuality quality, FrameSize frameSize, Boolean fec) [0x00000] in <filename unknown>:0
01-09 14:08:56.194  2142     1 E Unity   :   at Dissonance.CodecSettingsLoader.Generate () [0x00000] in <filename unknown>:0
01-09 14:08:56.194  2142     1 E Unity   :   at Dissonance.CodecSettingsLoader.get_Config () [0x00000] in <filename unknown>:0
01-09 14:08:56.194  2142     1 E Unity   :   at Dissonance.DissonanceComms.Start () [0x00000] in <filename unknown>:0
01-09 14:08:56.194  2142     1 E Unity   :   at System.Collections.Generic.IList`1[T].get_Item (Int32 index) [0x00000] in <filename unknown>:0
01-09 14:08:56.194  2142     1 E Unity   :   at UnityEngine.CSSLayout.Native..cctor () [0x00000] in <filename unknown>:0
01-09 14:08:56.194  2142     1 E Unity   : System.Collections.Generic.IList`1:get_Item(Int32)
01-09 14:08:56.194  2142     1 E Unity   : UnityEngine.CSSLayout.Native:.cctor()
01-09 14:08:56.194  2142     1 E Unity   :
01-09 14:08:56.194  2142     1 E Unity   : (Filename: currently not available on il2cpp Line: -1)`

and from shutdown

01-09 15:16:37.852  2685     4 D Unity   : UnityPerf: lifecycle on pause called at 36.352622 seconds
01-09 15:16:37.858  2685     1 E Unity   : Error: MagicLeapDevice.Update failed to get valid snapshot. Snapshot is NULL
01-09 15:16:37.858  2685     1 E Unity   : System.Diagnostics.StackTrace:.ctor(Int32, Boolean)
01-09 15:16:37.858  2685     1 E Unity   : UnityEngine.StackTraceUtility:ExtractStackTrace()
01-09 15:16:37.858  2685     1 E Unity   : UnityEngine.CSSLayout.Native:.cctor()
01-09 15:16:37.858  2685     1 E Unity   : UnityEngine.DebugLogHandler:Internal_Log(LogType, String, Object)
01-09 15:16:37.858  2685     1 E Unity   : UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
01-09 15:16:37.858  2685     1 E Unity   : UnityEngine.Logger:Log(LogType, Object)
01-09 15:16:37.858  2685     1 E Unity   : UnityEngine.Debug:LogError(Object)
01-09 15:16:37.858  2685     1 E Unity   : UnityEngine.XR.MagicLeap.MLPluginLog:Error(Object)
01-09 15:16:37.858  2685     1 E Unity   : UnityEngine.XR.MagicLeap.MagicLeapDevice:Update()
01-09 15:16:37.858  2685     1 E Unity   : HandyCollections.Heap.IMinHeap`1:IndexOf(Predicate`1)
01-09 15:16:37.858  2685     1 E Unity   : UnityEngine.CSSLayout.Native:.cctor()
01-09 15:16:37.858  2685     1 E Unity   :
01-09 15:16:37.858  2685     1 E Unity   : (Filename: C:/buildslave/unity/build\Runtime/Export/Debug.bindings.h Line: 43)
01-09 15:16:37.858  2685     1 E Unity   :
01-09 15:16:37.864  2685     1 I Unity   : Network layer will still work after application suspends
01-09 15:16:37.864  2685     1 I Unity   : System.Diagnostics.StackTrace:.ctor(Int32, Boolean)
01-09 15:16:37.864  2685     1 I Unity   : UnityEngine.StackTraceUtility:ExtractStackTrace()
01-09 15:16:37.864  2685     1 I Unity   : UnityEngine.CSSLayout.Native:.cctor()
01-09 15:16:37.864  2685     1 I Unity   :
01-09 15:16:37.864  2685     1 I Unity   : [C:/buildslave/unity/build/Runtime/Networking/UNETManager.cpp line 220]
01-09 15:16:37.864  2685     1 I Unity   : (Filename: C:/buildslave/unity/build/Runtime/Networking/UNETManager.cpp Line: 220)
01-09 15:16:37.864  2685     1 I Unity   :
01-09 15:16:41.974  2685    35 D Unity   : UnityPerf: lifecycle on stop called at 40.474970 seconds
01-09 15:16:42.022  2685     1 I Unity   : DEBUG [Dissonance:Core] (06:16:42.001) PlayerTrackerManager: Removed unlinked state tracker for 'd3d2e5c0-9dc7-4f7b-bd00-44142ed237c1' (because RemoveTracker called)
01-09 15:16:42.022  2685     1 I Unity   : System.Diagnostics.StackTrace:.ctor(Int32, Boolean)
01-09 15:16:42.022  2685     1 I Unity   : UnityEngine.StackTraceUtility:ExtractStackTrace()
01-09 15:16:42.022  2685     1 I Unity   : UnityEngine.CSSLayout.Native:.cctor()
01-09 15:16:42.022  2685     1 I Unity   : UnityEngine.DebugLogHandler:Internal_Log(LogType, String, Object)
01-09 15:16:42.022  2685     1 I Unity   : UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
01-09 15:16:42.022  2685     1 I Unity   : UnityEngine.Logger:Log(LogType, Object)
01-09 15:16:42.022  2685     1 I Unity   : UnityEngine.Debug:Log(Object)
01-09 15:16:42.022  2685     1 I Unity   : Dissonance.Logs:GetLogLevel(LogCategory)
01-09 15:16:42.022  2685     1 I Unity   : Dissonance.Logs:SendLogMessage(String, LogLevel)
01-09 15:16:42.022  2685     1 I Unity   : Dissonance.Log:WriteLog(LogLevel, String)
01-09 15:16:42.022  2685     1 I Unity   : Dissonance.Log:WriteLogFormat(LogLevel, String, TA)
01-09 15:16:42.022  2685     1 I Unity   : Dissonance.Log:Debug(String, TA)
01-09 15:16:42.022  2685     1 I Unity   : Dissonance.PlayerTrackerManager:RemoveTracker(IDissonancePlayer)
01-09 15:16:42.022  2685     1 I Unity   : Dissonance.DissonanceComms:StopTracking(IDissonancePlayer)
01-09 15:16:42.022  2685     1 I Unity   : Dissonance.Integrations.UNet_HLAPI.HlapiPlayer:StopTracking()
01-09 15:16:42.022  2685     1 I Unity   : Dissonance.Integrations.UNet_HLAPI.HlapiPlayer:OnDisable()
01-09 15:16:42.022  2685     1 I Unity   : HandyCollections.Heap.IMinHeap`1:IndexOf(Predicate`1)
01-09 15:16:42.022  2685     1 I Unity   : UnityEngine.CSSLayout.Native:.cctor()
01-09 15:16:42.022  2685     1 I Unity   :
01-09 15:16:42.022  2685     1 I Unity   : (Filename: C:/buildslave/unity/build\Runtime/Export/Debug.bindings.h Line: 43)
01-09 15:16:42.022  2685     1 I Unity   :
01-09 15:16:42.045  2685     1 I Unity   : DEBUG [Dissonance:Recording] (06:16:42.031) CapturePipelineManager: Destroying
01-09 15:16:42.045  2685     1 I Unity   : System.Diagnostics.StackTrace:.ctor(Int32, Boolean)
01-09 15:16:42.045  2685     1 I Unity   : UnityEngine.StackTraceUtility:ExtractStackTrace()
01-09 15:16:42.045  2685     1 I Unity   : UnityEngine.CSSLayout.Native:.cctor()
01-09 15:16:42.045  2685     1 I Unity   : UnityEngine.DebugLogHandler:Internal_Log(LogType, String, Object)
01-09 15:16:42.045  2685     1 I Unity   : UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
01-09 15:16:42.045  2685     1 I Unity   : UnityEngine.Logger:Log(LogType, Object)
01-09 15:16:42.045  2685     1 I Unity   : UnityEngine.Debug:Log(Object)
01-09 15:16:42.045  2685     1 I Unity   : Dissonance.Logs:GetLogLevel(LogCategory)
01-09 15:16:42.045  2685     1 I Unity   : Dissonance.Logs:SendLogMessage(String, LogLevel)
01-09 15:16:42.045  2685     1 I Unity   : Dissonance.Log:WriteLog(LogLevel, String)
01-09 15:16:42.045  2685     1 I Unity   : Dissonance.Log:Debug(String)
01-09 15:16:42.045  2685     1 I Unity   : Dissonance.Audio.Capture.CapturePipelineManager:Destroy()
01-09 15:16:42.045  2685     1 I Unity   : Dissonance.DissonanceComms:OnDestroy()
01-09 15:16:42.045  2685     1 I Unity   : HandyCollections.Heap.IMinHeap`1:IndexOf(Predicate`1)
01-09 15:16:42.045  2685     1 I Unity   : UnityEngine.CSSLayout.Native:.cctor()
01-09 15:16:42.045  2685     1 I Unity   :
01-09 15:16:42.045  2685     1 I Unity   : (Filename: C:/buildslave/unity/build\Runtime/Export/Debug.bindings.h Line: 43)
01-09 15:16:42.045  2685     1 I Unity   :
01-09 15:16:42.050  2685     1 E Unity   : NullReferenceException: A null value was found where an object instance was required.
01-09 15:16:42.050  2685     1 E Unity   :   at UnityEngine.ImageConversion.LoadImage (UnityEngine.Texture2D tex, System.Byte[] data) [0x00000] in <filename unknown>:0
01-09 15:16:42.050  2685     1 E Unity   :   at UnityEngine.JsonUtility.FromJson (System.String json, System.Type type) [0x00000] in <filename unknown>:0
01-09 15:16:42.050  2685     1 E Unity   :   at UnityEngine.JsonUtility.FromJson (System.String json, System.Type type) [0x00000] in <filename unknown>:0
01-09 15:16:42.050  2685     1 E Unity   :   at UnityEngine.JsonUtility.FromJson (System.String json, System.Type type) [0x00000] in <filename unknown>:0
01-09 15:16:42.050  2685     1 E Unity   :   at MagicLeap.ControllerStatusIndicator.OnDestroy () [0x00000] in <filename unknown>:0
01-09 15:16:42.050  2685     1 E Unity   :   at HandyCollections.Heap.IMinHeap`1[T].IndexOf (System.Predicate`1 predicate) [0x00000] in <filename unknown>:0
01-09 15:16:42.050  2685     1 E Unity   :   at UnityEngine.CSSLayout.Native..cctor () [0x00000] in <filename unknown>:0
01-09 15:16:42.050  2685     1 E Unity   : System.Diagnostics.StackTrace:.ctor(Int32, Boolean)
01-09 15:16:42.050  2685     1 E Unity   : UnityEngine.StackTraceUtility:ExtractStringFromExceptionInternal(Object, String&, String&)
01-09 15:16:42.050  2685     1 E Unity   : HandyCollections.Heap.IMinHeap`1:IndexOf(Predicate`1)
01-09 15:16:42.050  2685     1 E Unity   : UnityEngine.CSSLayout.Native:.cctor()
01-09 15:16:42.050  2685     1 E Unity   :
01-09 15:16:42.050  2685     1 E Unity   : (Filename: currently not available on il2cpp Line: -1)
01-09 15:16:42.050  2685     1 E Unity   :
01-09 15:16:42.055  2685     1 E Unity   : NullReferenceException: A null value was found where an object instance was required.
01-09 15:16:42.055  2685     1 E Unity   :   at UnityEngine.ImageConversion.LoadImage (UnityEngine.Texture2D tex, System.Byte[] data) [0x00000] in <filename unknown>:0
01-09 15:16:42.055  2685     1 E Unity   :   at UnityEngine.JsonUtility.FromJson (System.String json, System.Type type) [0x00000] in <filename unknown>:0
01-09 15:16:42.055  2685     1 E Unity   :   at UnityEngine.JsonUtility.FromJson (System.String json, System.Type type) [0x00000] in <filename unknown>:0
01-09 15:16:42.055  2685     1 E Unity   :   at UnityEngine.JsonUtility.FromJson (System.String json, System.Type type) [0x00000] in <filename unknown>:0
01-09 15:16:42.055  2685     1 E Unity   :   at MagicLeap.ControllerStatusText.OnDestroy () [0x00000] in <filename unknown>:0
01-09 15:16:42.055  2685     1 E Unity   :   at HandyCollections.Heap.IMinHeap`1[T].IndexOf (System.Predicate`1 predicate) [0x00000] in <filename unknown>:0
01-09 15:16:42.055  2685     1 E Unity   :   at UnityEngine.CSSLayout.Native..cctor () [0x00000] in <filename unknown>:0
01-09 15:16:42.055  2685     1 E Unity   : System.Diagnostics.StackTrace:.ctor(Int32, Boolean)
01-09 15:16:42.055  2685     1 E Unity   : UnityEngine.StackTraceUtility:ExtractStringFromExceptionInternal(Object, String&, String&)
01-09 15:16:42.055  2685     1 E Unity   : HandyCollections.Heap.IMinHeap`1:IndexOf(Predicate`1)
01-09 15:16:42.055  2685     1 E Unity   : UnityEngine.CSSLayout.Native:.cctor()

aethoco commented 5 years ago

We have multiple devices and are very keen on assisting with looking for a solution on the Leap. We too agree that it is an issue with the codec, anything we can do to help expedite testing and implementing a solution let us know.

Thanks for your debug output Frisky!

mechpil0t commented 5 years ago

OK I could not see the Dissonance log messages using mldb so I edited the Dissonance Log.cs script so that it would write all debug messages to a text file.

Here is the output:

[Dissonance:Core] (06:16:07.882) DissonanceComms: Loading default playback prefab:Info

DEBUG [Dissonance:Core] (06:16:14.278) PlayerTrackerManager: Got a player tracker for player 'd3d2e5c0-9dc7-4f7b-bd00-44142ed237c1' but that player doesn't exist yet:Debug

DEBUG [Dissonance:Network] (06:16:14.313) HLAPI Player Component: OnStartAuthority Name=cda581d9-7e7e-493d-a002-9f59e7500f84:Debug

DEBUG [Dissonance:Core] (06:16:14.336) PlayerTrackerManager: Associated position tracking for 'cda581d9-7e7e-493d-a002-9f59e7500f84':Debug

DEBUG [Dissonance:Core] (06:16:14.371) VoiceBroadcastTrigger: Collider entered (1):Debug

DEBUG [Dissonance:Core] (06:16:14.385) VoiceReceiptTrigger: Collider entered (1):Debug

DEBUG [Dissonance:Core] (06:16:14.408) VoiceReceiptTrigger: Recalculating token activation: 0 tokens, activated: True:Debug

DEBUG [Dissonance:Core] (06:16:14.422) Rooms: Joined chat room 'Room B':Debug

DEBUG [Dissonance:Core] (06:16:42.001) PlayerTrackerManager: Removed unlinked state tracker for 'd3d2e5c0-9dc7-4f7b-bd00-44142ed237c1' (because RemoveTracker called):Debug

DEBUG [Dissonance:Recording] (06:16:42.031) CapturePipelineManager: Destroying:Debug

Note (d3d2e5c0-9dc7-4f7b-bd00-44142ed237c1 was the ID of another device (a hololens) that was on the server) Some debug messages from that device:

DEBUG [Dissonance H:Core] (06:16:17.665) PlayerTrackerManager: Got a player tracker for player 'cda581d9-7e7e-493d-a002-9f59e7500f84' but that player doesn't exist yet
DEBUG [Dissonance H:Core] (06:16:47.303) PlayerTrackerManager: Removed unlinked state tracker for 'cda581d9-7e7e-493d-a002-9f59e7500f84' (because RemoveTracker called)

martindevans commented 5 years ago

Wow, thanks for the help guys!

Unable to find an entry point named 'dissonance_opus_encoder_ctl_in' in 'opus'.

It looks like this error may be quite simple to fix, it's not saying that the plugin file (opus.so) is incompatible, just that it can't find a specific entry point dissonance_opus_encoder_ctl_in. In Assets/Plugins/Dissonance/Core/Audio/Codecs/Opus/Opus.cs on line 93 that method is imported in this big ugly block of preprocessor directives:

#if UNITY_ANDROID && !UNITY_EDITOR
            [DllImport("opus", CallingConvention = CallingConvention.Cdecl)]
            internal static extern int opus_decoder_ctl(IntPtr st, Ctl request, int value);

            internal static int dissonance_opus_decoder_ctl_in(IntPtr st, Ctl request, int value)
            {
                return opus_decoder_ctl(st, request, value);
            }
#else
    #if UNITY_IOS && !UNITY_EDITOR
            [DllImport("__Internal", CallingConvention = CallingConvention.Cdecl)]
    #else
            [DllImport("opus", CallingConvention = CallingConvention.Cdecl)]
    #endif
            internal static extern int dissonance_opus_decoder_ctl_in(IntPtr st, Ctl request, int value);
#endif

You can see that there is special handling for Android - we import a different method from the plugin (opus_decoder_ctl) and wrap it in a method defined in C# (dissonance_opus_decoder_ctl_in). If you modify this so that magic leap is handled in the same way as Android it may fix the problem.

Unfortunately I can't see any MAGIC_LEAP directive available to the preprocessor (see docs here) so I'm not sure how best to do that (other than just defining your own directive).

Got a player tracker for player 'cda581d9-7e7e-493d-a002-9f59e7500f84' but that player doesn't exist yet Removed unlinked state tracker for 'cda581d9-7e7e-493d-a002-9f59e7500f84' (because RemoveTracker called)

This can happen if the player tracker for a player starts up before the player has joined the Dissonance session (which is quite common, the gameobject usually gets initialised before Dissonance has completed it's network handshake). The tracker gets stored in an unlinked state and when the player eventually joins the Dissonance session it gets linked up. The second one is a bit weird (it happens after 30 seconds, definitely enough time for someone to join) but let's not worry about it for now - session initialisation is probably being disrupted by the codec initialisation exploding!

aethoco commented 5 years ago

You rock @martindevans !

We checked it out and Magic Leap's directive is "PLATFORM_LUMIN".

We are about to test and report back shortly.

aethoco commented 5 years ago

@martindevans Magic Leap is now appearing as a peer, and we are able to receive audio from other instances that are Magic Leap, but still not able to send audio from the leap to other peers. We've gone through all the correct permissions on the Leap's side such as using the microphone, and we have gotten the microphone to work locally. Still having trouble sending that data over the network.

@FriskyBishop How were you able to output the debug log in your very first post? We can try and output the same now with our patches in place to see where things stand. Thanks guys!

martindevans commented 5 years ago

My guess is that the codec (opus.so) is fixed but the preprocessor (libAudioPluginDissonance.so) isn't - the preprocessor is significantly more complex so it may need to be recompiled against a magic leap specific set of libs. Definitely see if you can get a log file to confirm that.

If you go to Assets/Plugins/Dissonance/Core/Log.cs Line 53 you'll find a single method which all Dissonance logging goes through. You can probably just replace those Debug.Log calls with something like File.AppendAllText("log file path", log_msg); (that's not very performant, but it should serve as a temporary hack while we're debugging).

aethoco commented 5 years ago

Thanks @martindevans Let us attempt to generate the debug log and we will report back.

If it is an issue with the preprocessor needing to be recompiled, how can we assist in providing the right pieces to help achieve this goal? We are in communication now with Magic Leap as we try to resolve this issue.

martindevans commented 5 years ago

If you know anything about how to recompile C++ libs I can give you the source for all the native parts of Dissonance (Just email me (martin@placeholder-software.co.uk) your invoice number). That's going to be a lot quicker than trying to get me setup with ML development environment! I'm hoping it will be a relatively simple tweak to the current android build process (especially if you have support from ML themselves), feel free to copy me in on any discussions with them regarding the build process and I'll provide whatever tips I can.

aethoco commented 5 years ago

@martindevans We have an output:

01-09 15:42:17.064  4428     1 E Unity   : (Filename: currently not available on il2cpp Line: -1)
01-09 15:42:17.064  4428     1 E Unity   : 
01-09 15:42:17.078  4428     1 E Unity   : [Dissonance:Recording] (20:42:16.665) BasePreprocessingPipeline: Voice Error: Unhandled exception killed audio preprocessor thread: System.EntryPointNotFoundException: Unable to find an entry point named 'Dissonance_CreatePreprocessor' in 'AudioPluginDissonance'.
01-09 15:42:17.078  4428     1 E Unity   :   at UnityEngine.ImageConversion.LoadImage (UnityEngine.Texture2D tex, System.Byte[] data) [0x00000] in <filename unknown>:0 
01-09 15:42:17.078  4428     1 E Unity   :   at UnityEngine.JsonUtility.FromJson (System.String json, System.Type type) [0x00000] in <filename unknown>:0 
01-09 15:42:17.078  4428     1 E Unity   :   at UnityEngine.JsonUtility.FromJson (System.String json, System.Type type) [0x00000] in <filename unknown>:0 
01-09 15:42:17.078  4428     1 E Unity   :   at Dissonance.Audio.Capture.WebRtcPreprocessingPipeline+WebRtcPreprocessor.Dissonance_CreatePreprocessor (NoiseSuppressionLevels nsLevel, AecSuppressionLevels aecLevel, Boolean aecDelayAgnostic, Boolean aecExtended, Boolean aecRefined, AecmRoutingMode aecmRoutingMode, Boolean aecmComfortNoise) [0x00000] in <filename unknown>:0 
01-09 15:42:17.078  4428     1 E Unity   :   at Dissonance.Audio.Capture.WebRtcPreprocessingPipeline+WebRtcPreprocessor.CreatePreprocessor () [0x00000] in <filename unknown>:0 
01-09 15:42:17.078  4428     1 E Unity   :   at Dissonance.Audio.Capture.WebRtcPreprocessingPipeline+WebRtcPreprocessor.Reset () [0x00000] in <filename unknown>:0 
01-09 15:42:17.078  4428     1 E Unity   :   at Dissonance.Audio.Capture.WebRtcPreprocessingPipeline.ApplyReset () [0x00000] in <filename unknown>:0 
01-09 15:42:17.078  4428     1 E Unity   :   at Dissonance.Audio.Capture.BasePreprocessingPipeline.ThreadEntry () [0x00000] in <filename unknown>:0 
01-09 15:42:17.078  4428     1 E Unity   :   at System.Action.Invoke () [0x00000] in <filename unknown>:0 
01-09 15:42:17.078  4428     1 E Unity   :   at System.Threading.ThreadStart.Invoke () [0x00000] in <filename unknown>:0 
01-09 15:42:17.078  4428     1 E Unity   :   at Dissonance.IChannel`1[T].get_TargetId () [0x00000] in <filename unknown>:0 
01-09 15:42:17.078  4428     1 E Unity   :   at UnityEngine.CSSLayout.Native..cctor () [0x00000] in <filename unknown>:0 
01-09 15:42:17.078  4428     1 E Unity   :   at UnityEngine.CSSLayout.Native..cctor () [0x00000] in <filename unknown>:0 
01-09 15:42:17.078  4428     1 E Unity   :   at UnityEngine.ImageConversion.LoadImage (UnityEngine.Texture2D tex, System.Byte[] data) [0x00000] in <filename unknown>:0 
01-09 15:42:17.078  4428     1 E Unity   :   at UnityEngine.ImageConversion.LoadImage (UnityEngine.Texture2D tex, System.Byte[] data) [0x00000] in <filename unknown>:0 ! Error ID: 02EB75C0-1E12-4109-BFD2-64645C14BD5F
01-09 15:42:17.078  4428     1 E Unity   : System.Diagnostics.StackTrace:.ctor(Int32, Boolean)
01-09 15:42:17.078  4428     1 E Unity   : UnityEngine.StackTraceUtility:ExtractStackTrace()
01-09 15:42:17.078  4428     1 E Unity   : UnityEngine.CSSLayout.Native:.cctor()
01-09 15:42:17.078  4428     1 E Unity   : UnityEngine.DebugLogHandler:Internal_Log(LogType, String, Object)
01-09 15:42:17.078  4428     1 E Unity   : UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
01-09 15:42:17.078  4428     1 E Unity   : UnityEngine.Logger:Log(LogType, Object)
01-09 15:42:17.078  4428     1 E Unity   : UnityEngine.Debug:LogError(Object)
01-09 15:42:17.078  4428     1 E Unity   : Dissonance.Logs:GetLogLevel(LogCategory)
01-09 15:42:17.078  4428     1 E Unity   : Dissonance.Logs:WriteMultithreadedLogs()
01-09 15:42:17.078  4428     1 E Unity   : Dissonance.DissonanceComms:Update()
01-09 15:42:17.078  4428     1 E Unity   : Dissonance.IChannel`1:get_TargetId()
01-09 15:42:17.078  4428     1 E Unity   : UnityEngine.CSSLayout.Native:.cctor()

martindevans commented 5 years ago

Well that definitely confirms my suspicions about the preprocessor being the problem! Do you want to email me and take a stab at adapting the android build process yourself?

aethoco commented 5 years ago

@martindevans Shot you an email, thanks!

mechpil0t commented 5 years ago

@FriskyBishop How were you able to output the debug log in your very first post? We can try and output the same now with our patches in place to see where things stand. Thanks guys!

@aethoco ah i see you have a log - I was using mldb log https://creator.magicleap.com/learn/guides/sdk-mldb-debugging-apps

To get the debug log messages from dissonance to save to a text file I added the following to the Log.cs script ...

at line 93

        internal static void SendLogMessage(string message, LogLevel level)
        {
            string _l = level.ToString();
            Log.logit(message, _l);
#if NCRUNCH
            Console.WriteLine(message);
#else
            var msg = new LogMessage(message, level);

            if (_main == null || _main == Thread.CurrentThread)
                msg.Log();
            else
                LogsFromOtherThreads.TryWrite(msg);
#endif
        }
        #endregion
    }

then inside "public class Log" I added a simple write function

        public static void logit(string _a, string _b)
        {
          string folderName = Application.persistentDataPath + "/theLog.txt";
          if (!File.Exists(folderName))
          {
            File.Create(folderName).Dispose();
          }
          File.AppendAllText(folderName, _a + ":" + _b + Environment.NewLine);
        }

this should write a file to the /documents/C2 folder in the app directory

aethoco commented 5 years ago

@FriskyBishop Yup - we realized mldb with enabling the Dissonance debug was pretty straightforward after poking around a bit.

ML is looking internally to see if someone can help us compile the new preprocessor for Lumen OS. Will report back once we know more.

mechpil0t commented 5 years ago

@martindevans Magic Leap is now appearing as a peer, and we are able to receive audio from other instances that are Magic Leap, but still not able to send audio from the leap to other peers. We've gone through all the correct permissions on the Leap's side such as using the microphone, and we have gotten the microphone to work locally. Still having trouble sending that data over the network.

Can also confirm this I changed all the 'UNITY_ANDROID' to 'PLATFORM_LUMIN' in Opus.cs.

Magic Leap now appears as a peer and receives audio from other users.
Does not broadcast audio to other users.

Not sure if the logs will be helpful anymore but here we are ...

Here is the dissonance debug log:

[Dissonance:Core] (06:16:07.882) DissonanceComms: Loading default playback prefab:Info
DEBUG [Dissonance:Core] (06:16:14.278) PlayerTrackerManager: Got a player tracker for player 'd3d2e5c0-9dc7-4f7b-bd00-44142ed237c1' but that player doesn't exist yet:Debug
DEBUG [Dissonance:Network] (06:16:14.313) HLAPI Player Component: OnStartAuthority Name=cda581d9-7e7e-493d-a002-9f59e7500f84:Debug
DEBUG [Dissonance:Core] (06:16:14.336) PlayerTrackerManager: Associated position tracking for 'cda581d9-7e7e-493d-a002-9f59e7500f84':Debug
DEBUG [Dissonance:Core] (06:16:14.371) VoiceBroadcastTrigger: Collider entered (1):Debug
DEBUG [Dissonance:Core] (06:16:14.385) VoiceReceiptTrigger: Collider entered (1):Debug
DEBUG [Dissonance:Core] (06:16:14.408) VoiceReceiptTrigger: Recalculating token activation: 0 tokens, activated: True:Debug
DEBUG [Dissonance:Core] (06:16:14.422) Rooms: Joined chat room 'Room B':Debug
DEBUG [Dissonance:Core] (06:16:42.001) PlayerTrackerManager: Removed unlinked state tracker for 'd3d2e5c0-9dc7-4f7b-bd00-44142ed237c1' (because RemoveTracker called):Debug
DEBUG [Dissonance:Recording] (06:16:42.031) CapturePipelineManager: Destroying:Debug
[Dissonance:Core] (05:50:02.508) DissonanceComms: Loading default playback prefab:Info
DEBUG [Dissonance:Recording] (05:50:02.562) CapturePipelineManager: Not changing microphone device from '' to '' (equivalent devices):Debug
[Dissonance:Core] (05:50:02.579) DissonanceComms: Starting Dissonance Voice Comms
- Network: [DissonanceSetup (Dissonance.Integrations.UNet_HLAPI.HlapiCommsNetwork)]
- Quality Settings: [Quality: Low, FrameSize: Medium, FEC: True, DenoiseAmount: High, VoiceDuckLevel: 0.75]
- Codec: [Codec: Opus, FrameSize: 1920, SampleRate: 48kHz]:Info
DEBUG [Dissonance:Network] (05:50:08.857) HlapiCommsNetwork: Starting network session as Client:Debug
DEBUG [Dissonance:Recording] (05:50:08.893) CapturePipelineManager: Restarting transmission pipeline: 'Network mode changed':Debug
[Dissonance:Recording] (05:50:09.017) BasicMicrophoneCapture: Began mic capture (SampleRate:16000Hz, FrameSize:320, Buffer Limit:2^11, Latency:20ms, Device:''):Info
DEBUG [Dissonance:Recording] (05:50:09.035) BasePreprocessingPipeline: Started preprocessor thread. Internal latency: 0ms, Total stream latency:20ms:Debug
DEBUG [Dissonance:Recording] (05:50:09.145) BasicMicrophoneCapture: Trying to read 1024 samples, growing read buffer space to 2047:Debug
DEBUG [Dissonance:Core] (05:50:09.167) PlayerTrackerManager: Got a player tracker for player 'cb17aaaa-d633-4bf4-bef6-e352de9db844' but that player doesn't exist yet:Debug
DEBUG [Dissonance:Network] (05:50:09.200) HLAPI Player Component: OnStartAuthority Name=99a6faab-8120-4235-9d7b-4f8e30c314d0:Debug
DEBUG [Dissonance:Core] (05:50:09.224) PlayerTrackerManager: Associated position tracking for '99a6faab-8120-4235-9d7b-4f8e30c314d0':Debug
[Dissonance:Network] (05:50:09.255) ConnectionNegotiator`1: Received handshake response from server, joined session '556312583':Info
DEBUG [Dissonance:Network] (05:50:09.285) SlaveClientCollection`1: Sent local client state:Debug
DEBUG [Dissonance:Core] (05:50:09.318) VoiceBroadcastTrigger: Collider entered (1):Debug
DEBUG [Dissonance:Core] (05:50:09.332) VoiceReceiptTrigger: Collider entered (1):Debug
DEBUG [Dissonance:Core] (05:50:09.356) DissonanceComms: Net_PlayerJoined. playerId:cb17aaaa-d633-4bf4-bef6-e352de9db844 codecSettings:Codec: Opus, FrameSize: 1920, SampleRate: 48kHz:Debug
DEBUG [Dissonance:Core] (05:50:09.382) PlayerTrackerManager: Linked an unlinked player tracker for player 'cb17aaaa-d633-4bf4-bef6-e352de9db844':Debug
DEBUG [Dissonance:Playback] (05:50:09.409) SamplePlaybackComponent: Resetting playback component:Debug
DEBUG [Dissonance:Recording] (05:50:09.443) BasicMicrophoneCapture: Trying to read 3072 samples, growing read buffer space to 4095:Debug
DEBUG [Dissonance:Core] (05:50:09.457) VoiceReceiptTrigger: Recalculating token activation: 0 tokens, activated: True:Debug
DEBUG [Dissonance:Core] (05:50:09.471) Rooms: Joined chat room 'Room B':Debug
[Dissonance:Recording] (05:50:09.589) BasePreprocessingPipeline: Lost 320 samples in the preprocessor (buffer full), injecting silence to compensate:Warn

and his message was being spammed repeatedly in the mldb log:

01-10 14:50:18.129  1978     1 W Unity   : [Dissonance:Recording] (05:50:18.109) BasePreprocessingPipeline: Lost 320 samples in the preprocessor (buffer full), injecting silence to compensate
01-10 14:50:18.129  1978     1 W Unity   : System.Diagnostics.StackTrace:.ctor(Int32, Boolean)
01-10 14:50:18.129  1978     1 W Unity   : UnityEngine.StackTraceUtility:ExtractStackTrace()
01-10 14:50:18.129  1978     1 W Unity   : UnityEngine.CSSLayout.Native:.cctor()
01-10 14:50:18.129  1978     1 W Unity   : UnityEngine.DebugLogHandler:Internal_Log(LogType, String, Object)
01-10 14:50:18.129  1978     1 W Unity   : UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
01-10 14:50:18.129  1978     1 W Unity   : UnityEngine.Logger:Log(LogType, Object)
01-10 14:50:18.129  1978     1 W Unity   : UnityEngine.Debug:LogWarning(Object)
01-10 14:50:18.129  1978     1 W Unity   : Dissonance.Logs:GetLogLevel(LogCategory)
01-10 14:50:18.129  1978     1 W Unity   : Dissonance.Logs:SendLogMessage(String, LogLevel)
01-10 14:50:18.129  1978     1 W Unity   : Dissonance.Log:WriteLog(LogLevel, String)
01-10 14:50:18.129  1978     1 W Unity   : Dissonance.Log:WriteLogFormat(LogLevel, String, TA)
01-10 14:50:18.129  1978     1 W Unity   : Dissonance.Log:Warn(String, TA)
01-10 14:50:18.129  1978     1 W Unity   : Dissonance.Audio.Capture.BasePreprocessingPipeline:Dissonance.Audio.Capture.IMicrophoneSubscriber.ReceiveMicrophoneData(ArraySegment`1, WaveFormat)
01-10 14:50:18.129  1978     1 W Unity   : Dissonance.Audio.Capture.BasicMicrophoneCapture:SendFrame()
01-10 14:50:18.129  1978     1 W Unity   : Dissonance.Audio.Capture.BasicMicrophoneCapture:ConsumeSamples(ArraySegment`1)
01-10 14:50:18.129  1978     1 W Unity   : Dissonance.Audio.Capture.BasicMicrophoneCapture:DrainMicSamples()
01-10 14:50:18.129  1978     1 W Unity   : Dissonance.Audio.Capture.BasicMicrophoneCapture:UpdateSubscribers()
01-10 14:50:18.129  1978     1 W Unity   : Dissonance.Audio.Capture.CapturePipelineManager:Update(Boolean, Single)
01-10 14:50:18.129  1978     1 W Unity   : Dissonance.DissonanceComms:Update()
01-10 14:50:18.129  1978     1 W Unity   : System.Collections.Generic.ICollection`1:get_IsReadOnly()
01-10 14:50:18.129  1978     1 W Unity   : UnityEngine.CSSLayout.Native:.cctor()
01-10 14:50:18.129  1978     1 W Unity   :
01-10 14:50:18.129  1978     1 W Unity   : (Filename: C:/buildslave/unity/build\Runtime/Export/Debug.bindings.h Line: 43)

aethoco commented 5 years ago

Thanks @FriskyBishop I've been asking around, and it seems the best answer so far is that ML rolled their own: "The people I know that have done this so far all simply "rolled their own" solutions connecting our MLAudio API to a custom compression and network transport layer."

That said, if we can solve this issue, it would make Dissonance the de facto way to implement VOIP into their ML projects. Will keep you guys updated as we continue to hunt for assistance.

martindevans commented 5 years ago

@FriskyBishop That log is surprising, as far as I can see you never have the Unhandled exception killed audio preprocessor thread: System.EntryPointNotFoundException: Unable to find an entry point named 'Dissonance_CreatePreprocessor' in 'AudioPluginDissonance'. message?

@aethoco I've sent you the source code by email. I'll try to make some time next week to investigate into building it for ML.

connecting our MLAudio API to a custom compression and network transport layer

I know I may be slightly biased but it's really not that simple! Just stuffing audio into a network stream handles none of the cases required for good audio quality with bad network conditions >_<

aethoco commented 5 years ago

A dev at ML has been reviewing this thread and suggested the following:

"I'm looking at the error log posted on the GitHub issue and this may be a different problem.. The error log says it's trying to consume a 16khz stream, but our World CAmera stream only support 48khz.

So everything may be working, but you may be getting silence from the mics. (Looking at MLAudioCreateInputFromWorldCapture in the API)"

martindevans commented 5 years ago

We do have this System.EntryPointNotFoundException which will definitely kill the audio encoding system. I don't think that can be caused by an incorrect sample rate. @FriskyBishop doesn't seem to have that which is odd - did either of you change anything else (e.g. plugin import settings)?

The 16kHz sample rate being referred to is presumably from this log message:

[Dissonance:Recording] (05:50:09.017) BasicMicrophoneCapture: Began mic capture (SampleRate:16000Hz, FrameSize:320, Buffer Limit:2^11, Latency:20ms, Device:''):Info

The sample rate is chosen by this code (Assets/Plugins/Dissonance/Core/Audio/Capture/BasicMicrophoneCapture.cs Line 64):

Microphone.GetDeviceCaps(_micName, out minFreq, out maxFreq);
var sampleRate = minFreq == 0 && maxFreq == 0 ? 48000 : Mathf.Clamp(48000, minFreq, maxFreq);

GetDeviceCaps returns min:0 max:0 if the device can handle any rate, in which case we default to 48kHz. Otherwise we select the closest value to 48kHz within the valid range.

If we're selecting 16,000 then GetDeviceCaps has returned the wrong limits. Could you check what values this returns:

int minFreq, maxFreq;
Microphone.GetDeviceCaps("", out minFreq, out maxFreq);

If the ML only support 48kHz then both these values should be 48,000. If they're not we can report a bug to someone.

As a workaround you could just hardcode 48kHz as the sample rate here.

#if PLATFORM_LUMIN
    var sampleRate = 48000;
#else
    Microphone.GetDeviceCaps(_micName, out minFreq, out maxFreq);
    var sampleRate = minFreq == 0 && maxFreq == 0 ? 48000 : Mathf.Clamp(48000, minFreq, maxFreq);
#endif

aethoco commented 5 years ago

@martindevans Thanks Martin. We will attempt the 48khz hardcode patch this afternoon and let everyone know.

aethoco commented 5 years ago

@martindevans We attempted the 48khz hardcode patch and still at the same result, our debug logs:

First message proves we are capturing the mic at 48000:

01-11 09:01:18.045 1585 1 I Unity : [Dissonance:Recording] (14:01:18.026) BasicMicrophoneCapture: Began mic capture (SampleRate:48000Hz, FrameSize:960, Buffer Limit:2^13, Latency:20ms, Device:'') 01-11 09:01:18.045 1585 1 I Unity : System.Diagnostics.StackTrace:.ctor(Int32, Boolean) 01-11 09:01:18.045 1585 1 I Unity : UnityEngine.StackTraceUtility:ExtractStackTrace() 01-11 09:01:18.045 1585 1 I Unity : UnityEngine.CSSLayout.Native:.cctor() 01-11 09:01:18.045 1585 1 I Unity : UnityEngine.DebugLogHandler:Internal_Log(LogType, String, Object) 01-11 09:01:18.045 1585 1 I Unity : UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[]) 01-11 09:01:18.045 1585 1 I Unity : UnityEngine.Logger:Log(LogType, Object) 01-11 09:01:18.045 1585 1 I Unity : UnityEngine.Debug:Log(Object) 01-11 09:01:18.045 1585 1 I Unity : Dissonance.Logs:GetLogLevel(LogCategory) 01-11 09:01:18.045 1585 1 I Unity : Dissonance.Logs:SendLogMessage(String, LogLevel) 01-11 09:01:18.045 1585 1 I Unity : Dissonance.Log:WriteLog(LogLevel, String) 01-11 09:01:18.045 1585 1 I Unity : Dissonance.Log:WriteLogFormat(LogLevel, String, TA, TB, TC, TD, TE) 01-11 09:01:18.045 1585 1 I Unity : Dissonance.Log:Info(String, TA, TB, TC, TD, TE) 01-11 09:01:18.045 1585 1 I Unity : Dissonance.Audio.Capture.BasicMicrophoneCapture:StartCapture(String) 01-11 09:01:18.045 1585 1 I Unity : Dissonance.Audio.Capture.CapturePipelineManager:RestartTransmissionPipeline(String) 01-11 09:01:18.045 1585 1 I Unity : Dissonance.Audio.Capture.CapturePipelineManager:Update(Boolean, Single) 01-11 09:01:18.045 1585 1 I Unity : Dissonance.DissonanceComms:Update() 01-11 09:01:18.045 1585 1 I Unity : UnityEngine.ImageConversion:LoadImage(Texture2D, Byte[]) 01-11 09:01:18.045 1585 1 I Unity : UnityEngine.CSSLayout.Native:.cctor()

Second message proves we are stuck at the preprocessor:

01-11 09:01:18.337 1585 1 E Unity : [Dissonance:Recording] (14:01:18.051) BasePreprocessingPipeline: Voice Error: Unhandled exception killed audio preprocessor thread: System.EntryPointNotFoundException: Unable to find an entry point named 'Dissonance_CreatePreprocessor' in 'AudioPluginDissonance'.

Third message we are being spammed with at every frame - Is this message being sent due to the preprocessor error?

01-11 09:01:18.972 1585 1 W Unity : [Dissonance:Recording] (14:01:18.951) BasePreprocessingPipeline: Lost 960 samples in the preprocessor (buffer full), injecting silence to compensate

Thanks!

martindevans commented 5 years ago

Third message we are being spammed with at every frame

That's caused by the earlier exception. We have the main Unity thread putting raw mic audio into the input buffer and the background preprocessor thread draining the buffer... but the background thread is dead so the input buffer is permanently full and so audio is being discarded.

BasePreprocessingPipeline: Voice Error: Unhandled exception killed audio preprocessor thread

I'm really interested why @FriskyBishop doesn't seem to get this exception. Could you (@FriskyBishop) go to Assets/Plugins/Dissonance/Core/Audio/Capture/WebRtcPreprocessingPipeline.cs Line 310 and put in some log messages:

Debug.Log("--> About to create preprocessor");
var preprocessor = Dissonance_CreatePreprocessor(
                    NoiseSuppressionLevel,
                    pcLevel,
                    instance.AecDelayAgnostic,
                    instance.AecExtendedFilter,
                    instance.AecRefinedAdaptiveFilter,
                    mobLevel,
                    instance.AecmComfortNoise
                );
Debug.Log(string.Format("--> Created Preprocessor: {0}", preprocessor));
return preprocessor

If that prints both messages (when run on the Magic Leap) then we know the AudioPluginDissonance.so plugin is loaded and the Dissonance_CreatePreprocessor entry point is being found.

Did either of your change the import settings for the files found in https://github.com/TomGillen/Dissonance/tree/master/Assets/Plugins/Dissonance/Plugins/Android/libs/armeabi-v7a ?

aethoco commented 5 years ago

@martindevans We did a complete install of Dissonance without doing anything other than prompted by the Getting Started Guide.

However this might be it on our end - we're building specifically for Lumin.

https://i.imgur.com/sCOa44O.jpg

Lumen was unticked

martindevans commented 5 years ago

That definitely should be ticked. I must admit I'm a little confused how opus is working if that isn't checked on opus.so 😕

aethoco commented 5 years ago

Ticking Lumen, the app builds fine - with no errors - but I can't deploy the app on the device - I click the app and it just kicks me back to the main menu every time.

To add: The ML debug log output nothing

martindevans commented 5 years ago

Can you ask your magic leap contact what they think of that? It seems odd that the program dies with nothing at all written to the log. If possible I'd like to get confirmation that's the behaviour they'd expect for an incompatible binary plugin.

My plan at the moment is to download the Lumin SDK on Monday and see if I can adapt the android build process for AudioPluginDissonance.so to simply use the Lumin SDK instead of the android SDK.

aethoco commented 5 years ago

@martindevans Will do.

Good luck for Monday, let us know if we can help test as you move through that process.

martindevans commented 5 years ago

I've managed to get the plugin to build with the mabu build tool. This initially threw some Magic Leap specific errors which reassures me that it was actually building for the right thing! I've emailed the build to @aethoco so we'll see if this actually works on the device!

mechpil0t commented 5 years ago

@martindevans like @aethoco I did not check Lumen on any of the plugins I tried checking the one you suggested, it builds and installs on the device but will not run.

I will add the debug code and test today

Could you email the build to me too? I'd like to test it matt@thedesignium.com

martindevans commented 5 years ago

@FriskyBishop I've forwarded you the email I sent to @aethoco earlier today with my test build attached. Make sure to hardcode the 48kHz change as mentioned in this post when you're testing it :)

mechpil0t commented 5 years ago

I added the debug to the WebRtcPreprocessingPipeline.cs

Debug.Log("--> About to create preprocessor");
var preprocessor = Dissonance_CreatePreprocessor(
                    NoiseSuppressionLevel,
                    pcLevel,
                    instance.AecDelayAgnostic,
                    instance.AecExtendedFilter,
                    instance.AecRefinedAdaptiveFilter,
                    mobLevel,
                    instance.AecmComfortNoise
                );
Debug.Log(string.Format("--> Created Preprocessor: {0}", preprocessor));
return preprocessor

and tested the new plugin ...

Same results as before - Magic Leap can receive but is not sending data

I edited the basic mic script

I added

#if PLATFORM_LUMIN
    var sampleRate = 48000;
#else
    Microphone.GetDeviceCaps(_micName, out minFreq, out maxFreq);
    var sampleRate = minFreq == 0 && maxFreq == 0 ? 48000 : Mathf.Clamp(48000, minFreq, maxFreq);
#endif

at line 64 of BasicMicrophoneCapture.cs

but maybe it didnt work or I edited it wrong as I get alot of debug errors from BasePreprocessingPipeline and BasicMicrophoneCapture

On start-up

01-15 10:06:28.408  2087     1 I Unity   : [Dissonance:Core] (01:06:28.394) DissonanceComms: Loading default playback prefab
01-15 10:06:28.408  2087     1 I Unity   : System.Diagnostics.StackTrace:.ctor(Int32, Boolean)
01-15 10:06:28.408  2087     1 I Unity   : UnityEngine.StackTraceUtility:ExtractStackTrace()
01-15 10:06:28.408  2087     1 I Unity   : UnityEngine.CSSLayout.Native:.cctor()
01-15 10:06:28.408  2087     1 I Unity   : UnityEngine.DebugLogHandler:Internal_Log(LogType, String, Object)
01-15 10:06:28.408  2087     1 I Unity   : UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
01-15 10:06:28.408  2087     1 I Unity   : UnityEngine.Logger:Log(LogType, Object)
01-15 10:06:28.408  2087     1 I Unity   : UnityEngine.Debug:Log(Object)
01-15 10:06:28.408  2087     1 I Unity   : Dissonance.Logs:GetLogLevel(LogCategory)
01-15 10:06:28.408  2087     1 I Unity   : Dissonance.Logs:SendLogMessage(String, LogLevel)
01-15 10:06:28.408  2087     1 I Unity   : Dissonance.Log:WriteLog(LogLevel, String)
01-15 10:06:28.408  2087     1 I Unity   : Dissonance.Log:Info(String)
01-15 10:06:28.408  2087     1 I Unity   : Dissonance.DissonanceComms:Start()
01-15 10:06:28.408  2087     1 I Unity   : UnityEngine.ImageConversion:LoadImage(Texture2D, Byte[])
01-15 10:06:28.408  2087     1 I Unity   : UnityEngine.CSSLayout.Native:.cctor()
01-15 10:06:28.408  2087     1 I Unity   :
01-15 10:06:28.408  2087     1 I Unity   : (Filename: C:/buildslave/unity/build\Runtime/Export/Debug.bindings.h Line: 43)
01-15 10:06:28.408  2087     1 I Unity   :
01-15 10:06:28.415  2087     1 D Unity   : Looking for Plugin opus -> /package/bin
01-15 10:06:28.415  2087     1 D Unity   : Looking for Plugin opus -> /system/lib64
01-15 10:06:28.464  2087     1 I Unity   : DEBUG [Dissonance:Recording] (01:06:28.450) CapturePipelineManager: Not changing microphone device from '' to '' (equivalent devices)
01-15 10:06:28.464  2087     1 I Unity   : System.Diagnostics.StackTrace:.ctor(Int32, Boolean)
01-15 10:06:28.464  2087     1 I Unity   : UnityEngine.StackTraceUtility:ExtractStackTrace()
01-15 10:06:28.464  2087     1 I Unity   : UnityEngine.CSSLayout.Native:.cctor()
01-15 10:06:28.464  2087     1 I Unity   : UnityEngine.DebugLogHandler:Internal_Log(LogType, String, Object)
01-15 10:06:28.464  2087     1 I Unity   : UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
01-15 10:06:28.464  2087     1 I Unity   : UnityEngine.Logger:Log(LogType, Object)
01-15 10:06:28.464  2087     1 I Unity   : UnityEngine.Debug:Log(Object)
01-15 10:06:28.464  2087     1 I Unity   : Dissonance.Logs:GetLogLevel(LogCategory)
01-15 10:06:28.464  2087     1 I Unity   : Dissonance.Logs:SendLogMessage(String, LogLevel)
01-15 10:06:28.464  2087     1 I Unity   : Dissonance.Log:WriteLog(LogLevel, String)
01-15 10:06:28.464  2087     1 I Unity   : Dissonance.Log:WriteLogFormat(LogLevel, String, TA, TB)
01-15 10:06:28.464  2087     1 I Unity   : Dissonance.Log:Debug(String, TA, TB)
01-15 10:06:28.464  2087     1 I Unity   : Dissonance.Audio.Capture.CapturePipelineManager:set_MicrophoneName(String)
01-15 10:06:28.464  2087     1 I Unity   : Dissonance.DissonanceComms:Start()
01-15 10:06:28.464  2087     1 I Unity   : UnityEngine.ImageConversion:LoadImage(Texture2D, Byte[])
01-15 10:06:28.464  2087     1 I Unity   : UnityEngine.CSSLayout.Native:.cctor()
01-15 10:06:28.464  2087     1 I Unity   :
01-15 10:06:28.464  2087     1 I Unity   : (Filename: C:/buildslave/unity/build\Runtime/Export/Debug.bindings.h Line: 43)
01-15 10:06:28.464  2087     1 I Unity   :
01-15 10:06:28.480  2087     1 I Unity   : [Dissonance:Core] (01:06:28.467) DissonanceComms: Starting Dissonance Voice Comms
01-15 10:06:28.480  2087     1 I Unity   : - Network: [DissonanceSetup (Dissonance.Integrations.UNet_HLAPI.HlapiCommsNetwork)]
01-15 10:06:28.480  2087     1 I Unity   : - Quality Settings: [Quality: Low, FrameSize: Medium, FEC: True, DenoiseAmount: High, VoiceDuckLevel: 0.75]
01-15 10:06:28.480  2087     1 I Unity   : - Codec: [Codec: Opus, FrameSize: 1920, SampleRate: 48kHz]
01-15 10:06:28.480  2087     1 I Unity   : System.Diagnostics.StackTrace:.ctor(Int32, Boolean)
01-15 10:06:28.480  2087     1 I Unity   : UnityEngine.StackTraceUtility:ExtractStackTrace()
01-15 10:06:28.480  2087     1 I Unity   : UnityEngine.CSSLayout.Native:.cctor()
01-15 10:06:28.480  2087     1 I Unity   : UnityEngine.DebugLogHandler:Internal_Log(LogType, String, Object)
01-15 10:06:28.480  2087     1 I Unity   : UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
01-15 10:06:28.480  2087     1 I Unity   : UnityEngine.Logger:Log(LogType, Object)
01-15 10:06:28.480  2087     1 I Unity   : UnityEngine.Debug:Log(Object)
01-15 10:06:28.480  2087     1 I Unity   : Dissonance.Logs:GetLogLevel(LogCategory)
01-15 10:06:28.480  2087     1 I Unity   : Dissonance.Logs:SendLogMessage(String, LogLevel)
01-15 10:06:28.480  2087     1 I Unity   : Dissonance.Log:WriteLog(LogLevel, String)
01-15 10:06:28.480  2087     1 I Unity   : Dissonance.Log:WriteLogFormat(LogLevel, String, TA, TB, TC)
01-15 10:06:28.480  2087     1 I Unity   : Dissonance.Log:Info(String, TA, TB, TC)
01-15 10:06:28.480  2087     1 I Unity   : Dissonance.DissonanceComms:Start()
01-15 10:06:28.480  2087     1 I Unity   : UnityEngine.ImageConversion:LoadImage(Texture2D, Byte[])
01-15 10:06:28.480  2087     1 I Unity   : UnityEngine.CSSLayout.Native:.cctor()
01-15 10:06:28.480  2087     1 I Unity   :
01-15 10:06:28.480  2087     1 I Unity   : (Filename: C:/buildslave/unity/build\Runtime/Export/Debug.bindings.h Line: 43)
01-15 10:06:28.480  2087     1 I Unity   :
01-15 10:06:28.486  2087     1 D Unity   : Looking for Plugin ml_platform -> /package/bin
01-15 10:06:28.486  2087     1 D Unity   : Looking for Plugin ml_platform -> /system/lib64
01-15 10:06:28.498  2087     1 E Unity   : Error: MagicLeapDevice.Update failed to get valid snapshot. Snapshot is NULL
01-15 10:06:28.498  2087     1 E Unity   : System.Diagnostics.StackTrace:.ctor(Int32, Boolean)
01-15 10:06:28.498  2087     1 E Unity   : UnityEngine.StackTraceUtility:ExtractStackTrace()
01-15 10:06:28.498  2087     1 E Unity   : UnityEngine.CSSLayout.Native:.cctor()
01-15 10:06:28.498  2087     1 E Unity   : UnityEngine.DebugLogHandler:Internal_Log(LogType, String, Object)
01-15 10:06:28.498  2087     1 E Unity   : UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
01-15 10:06:28.498  2087     1 E Unity   : UnityEngine.Logger:Log(LogType, Object)
01-15 10:06:28.498  2087     1 E Unity   : UnityEngine.Debug:LogError(Object)
01-15 10:06:28.498  2087     1 E Unity   : UnityEngine.XR.MagicLeap.MLPluginLog:Error(Object)
01-15 10:06:28.498  2087     1 E Unity   : UnityEngine.XR.MagicLeap.MagicLeapDevice:Update()
01-15 10:06:28.498  2087     1 E Unity   : UnityEngine.ImageConversion:LoadImage(Texture2D, Byte[])
01-15 10:06:28.498  2087     1 E Unity   : UnityEngine.CSSLayout.Native:.cctor()

then is spams the console with this:

01-15 10:09:40.961  2087     1 I Unity   : System.Diagnostics.StackTrace:.ctor(Int32, Boolean)
01-15 10:09:40.961  2087     1 I Unity   : UnityEngine.StackTraceUtility:ExtractStackTrace()
01-15 10:09:40.961  2087     1 I Unity   : UnityEngine.CSSLayout.Native:.cctor()
01-15 10:09:40.961  2087     1 I Unity   : UnityEngine.DebugLogHandler:Internal_Log(LogType, String, Object)
01-15 10:09:40.961  2087     1 I Unity   : UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
01-15 10:09:40.961  2087     1 I Unity   : UnityEngine.Logger:Log(LogType, Object)
01-15 10:09:40.961  2087     1 I Unity   : UnityEngine.Debug:Log(Object)
01-15 10:09:40.961  2087     1 I Unity   : Dissonance.Logs:GetLogLevel(LogCategory)
01-15 10:09:40.961  2087     1 I Unity   : Dissonance.Logs:SendLogMessage(String, LogLevel)
01-15 10:09:40.961  2087     1 I Unity   : Dissonance.Log:WriteLog(LogLevel, String)
01-15 10:09:40.961  2087     1 I Unity   : Dissonance.Log:Debug(String)
01-15 10:09:40.961  2087     1 I Unity   : Dissonance.Audio.Capture.CapturePipelineManager:Destroy()
01-15 10:09:40.961  2087     1 I Unity   : Dissonance.DissonanceComms:OnDestroy()
01-15 10:09:40.961  2087     1 I Unity   : UnityEngine.ImageConversion:LoadImage(Texture2D, Byte[])
01-15 10:09:40.961  2087     1 I Unity   : UnityEngine.CSSLayout.Native:.cctor()
01-15 10:09:40.961  2087     1 I Unity   :
01-15 10:09:40.961  2087     1 I Unity   : (Filename: C:/buildslave/unity/build\Runtime/Export/Debug.bindings.h Line: 43)
01-15 10:09:40.961  2087     1 I Unity   :

Logs from Dissonance Debug:

[Dissonance:Core] (01:06:28.394) DissonanceComms: Loading default playback prefab:Info
DEBUG [Dissonance:Recording] (01:06:28.450) CapturePipelineManager: Not changing microphone device from '' to '' (equivalent devices):Debug
[Dissonance:Core] (01:06:28.467) DissonanceComms: Starting Dissonance Voice Comms
- Network: [DissonanceSetup (Dissonance.Integrations.UNet_HLAPI.HlapiCommsNetwork)]
- Quality Settings: [Quality: Low, FrameSize: Medium, FEC: True, DenoiseAmount: High, VoiceDuckLevel: 0.75]
- Codec: [Codec: Opus, FrameSize: 1920, SampleRate: 48kHz]:Info
DEBUG [Dissonance:Network] (01:06:34.758) HlapiCommsNetwork: Starting network session as Client:Debug
DEBUG [Dissonance:Recording] (01:06:34.793) CapturePipelineManager: Restarting transmission pipeline: 'Network mode changed':Debug
[Dissonance:Recording] (01:06:34.987) BasicMicrophoneCapture: Began mic capture (SampleRate:48000Hz, FrameSize:960, Buffer Limit:2^13, Latency:20ms, Device:''):Info
DEBUG [Dissonance:Recording] (01:06:35.005) BasePreprocessingPipeline: Started preprocessor thread. Internal latency: 0ms, Total stream latency:20ms:Debug
DEBUG [Dissonance:Core] (01:06:35.119) PlayerTrackerManager: Got a player tracker for player '8a65b737-e920-4701-b17d-d7c9656b4131' but that player doesn't exist yet:Debug
DEBUG [Dissonance:Network] (01:06:35.154) HLAPI Player Component: OnStartAuthority Name=370a9ec1-3321-42e8-94b8-20bb872636c6:Debug
DEBUG [Dissonance:Core] (01:06:35.181) PlayerTrackerManager: Associated position tracking for '370a9ec1-3321-42e8-94b8-20bb872636c6':Debug
[Dissonance:Network] (01:06:35.215) ConnectionNegotiator`1: Received handshake response from server, joined session '912044613':Info
DEBUG [Dissonance:Network] (01:06:35.248) SlaveClientCollection`1: Sent local client state:Debug
DEBUG [Dissonance:Core] (01:06:35.285) VoiceBroadcastTrigger: Collider entered (1):Debug
DEBUG [Dissonance:Core] (01:06:35.300) VoiceReceiptTrigger: Collider entered (1):Debug
DEBUG [Dissonance:Core] (01:06:35.326) DissonanceComms: Net_PlayerJoined. playerId:8a65b737-e920-4701-b17d-d7c9656b4131 codecSettings:Codec: Opus, FrameSize: 1920, SampleRate: 48kHz:Debug
DEBUG [Dissonance:Core] (01:06:35.357) PlayerTrackerManager: Linked an unlinked player tracker for player '8a65b737-e920-4701-b17d-d7c9656b4131':Debug
DEBUG [Dissonance:Playback] (01:06:35.390) SamplePlaybackComponent: Resetting playback component:Debug
DEBUG [Dissonance:Recording] (01:06:35.432) BasicMicrophoneCapture: Trying to read 14880 samples, growing read buffer space to 2047:Debug
DEBUG [Dissonance:Recording] (01:06:35.450) BasicMicrophoneCapture: Trying to read 14880 samples, growing read buffer space to 4095:Debug
DEBUG [Dissonance:Recording] (01:06:35.467) BasicMicrophoneCapture: Trying to read 14880 samples, growing read buffer space to 8191:Debug
DEBUG [Dissonance:Recording] (01:06:35.484) BasicMicrophoneCapture: Trying to read 14880 samples, growing read buffer space to 16383:Debug
[Dissonance:Recording] (01:06:35.501) BasePreprocessingPipeline: Lost 960 samples in the preprocessor (buffer full), injecting silence to compensate:Warn
[Dissonance:Recording] (01:06:35.522) BasePreprocessingPipeline: Lost 960 samples in the preprocessor (buffer full), injecting silence to compensate:Warn
[Dissonance:Recording] (01:06:35.545) BasePreprocessingPipeline: Lost 960 samples in the preprocessor (buffer full), injecting silence to compensate:Warn
[Dissonance:Recording] (01:06:35.566) BasePreprocessingPipeline: Lost 960 samples in the preprocessor (buffer full), injecting silence to compensate:Warn
[Dissonance:Recording] (01:06:35.587) BasePreprocessingPipeline: Lost 960 samples in the preprocessor (buffer full), injecting silence to compensate:Warn
[Dissonance:Recording] (01:06:35.608) BasePreprocessingPipeline: Lost 960 samples in the preprocessor (buffer full), injecting silence to compensate:Warn
[Dissonance:Recording] (01:06:35.629) BasePreprocessingPipeline: Lost 960 samples in the preprocessor (buffer full), injecting silence to compensate:Warn
DEBUG [Dissonance:Core] (01:06:35.652) VoiceReceiptTrigger: Recalculating token activation: 0 tokens, activated: True:Debug
DEBUG [Dissonance:Core] (01:06:35.668) Rooms: Joined chat room 'Room B':Debug

then spams: [Dissonance:Recording] (01:06:36.310) BasePreprocessingPipeline: Lost 960 samples in the preprocessor (buffer full), injecting silence to compensate:Warn

Then I get:

DEBUG [Dissonance:Playback] (01:06:36.379) SpeechSessionStream: Created new speech session with buffer time of 100ms:Debug
[Dissonance:Recording] (01:06:36.413) BasicMicrophoneCapture: Insufficient buffer space, requested 18240, clamped to 16383 (dropping samples):Warn
[Dissonance:Recording] (01:06:37.269) BasicMicrophoneCapture: Insufficient buffer space, requested 28674, clamped to 16383 (dropping samples):Warn

martindevans commented 5 years ago

@FriskyBishop Those log snippets don't show the --> About to create preprocessor or --> Created Preprocessor: messages. Were those messages in the full log and just didn't get included in these snippets, or are those events genuinely missing? The message being spammed to console is very odd, the root of that stacktrace is Dissonance.DissonanceComms:OnDestroy() which implies Unity is (repeatedly) destroying the component?! Can you attach another component to the same gameObject which logs Unity lifecycle events, so see if Unity is doing something odd.

@aethoco Do you see similar results?

aethoco commented 5 years ago

@martindevans We should have time to test later this evening, will report back then.

aethoco commented 5 years ago

@martindevans Didn't work, confirmed initialized mic with correct sample rate - just still getting spammed with these messages whenever talking:

01-15 15:47:06.275 4385 1 W Unity : [Dissonance:Recording] (20:47:06.260) BasicMicrophoneCapture: Insufficient buffer space, requested 111048, clamped to 16383 (dropping samples)

01-15 15:47:06.294 4385 1 W Unity : [Dissonance:Recording] (20:47:06.275) BasePreprocessingPipeline: Lost 480 samples in the preprocessor (buffer full), injecting silence to compensate

martindevans commented 5 years ago

Are there any errors like Unhandled exception killed audio preprocessor thread: <details of error>? If not, could you check a few other things:

Assets/Plugins/Dissonance/Core/Audio/Capture/WebRtcPreprocessingPipeline.cs Line 248 public bool Process... Insert a log message in here on entry and exit (print inputSampleRate, input.length and output.Length just to double check). This should get printed for every single frame of audio to be preprocessed. this will confirm that the preprocessor is truly running and processing audio as expected. If you also log the exact timestamp we can check that the expected number of samples per second are being processed.
Assets/Plugins/Dissonance/Core/Audio/Capture/BasicMicrophoneCapture.cs Line 328 //Send frame to subscribers Insert a log message here showing how many samples are being sent (segment.Count) and to how many subscribers (_subscribers.Count). This will be printed for every frame out audio produced by the mic and will confirm that the expected data is being recorded. Again if you log the timestamp we can verify that the correct number of samples are being produced.
Enable recording diagnostics (Window > Dissonance > Diagnostic Setting) for all the capture parts of the pipeline. This will record several files to disk with raw audio captured from various stages of the pipeline. The file location will be something like Dissonance_Diagnostics/MicrophoneRawAudio_<TimeStamp>. If you need to change that search for all uses of AudioFileWriter and substitute in whatever path you want.

aethoco commented 5 years ago

Are there any errors like Unhandled exception killed audio preprocessor thread:
? If not, could you check a few other things:

There were no errors of this type. We will try out the rest of the requests tomorrow and circle back. Thanks!

aethoco commented 5 years ago

Update, we have not gotten time to sort through those requests yet. @FriskyBishop have you had any luck so far?

mechpil0t commented 5 years ago

Update, we have not gotten time to sort through those requests yet. @FriskyBishop have you had any luck so far?

@aethoco unfortunately I have not had time yet either - hopefully I can give it another try next week

aethoco commented 5 years ago

Unfortunately we have not had time to check this out in detail, as we have pressing projects that have to go out the door. Hoping @FriskyBishop might have some time, as he's been smarter than us at debugging things so far. :)

Hopefully in a week or two we might have a day or two to spend debugging. @martindevans If I were in the UK I'd loan you a Leap if I could at this point!

mechpil0t commented 5 years ago

Our magic leap has been out of office with another employee for a week or so >.< so I have not had chance to test anymore - however - news from the Magic Leap forums

martindevans commented 5 years ago

Magic Leap are sending a device my way, hopefully that will arrive within a week or two and I can do all the testing myself. They're also going to get an engineer to work with me next week to help with porting Dissonance.

aethoco commented 5 years ago

Great to hear @martindevans ! Glad to see the massive effort ML is going through to help resolve this, big thanks to you and their team for the push. Also, enjoy playing with the MLO, it's a real eye opener :)

martindevans commented 5 years ago

Dissonance 6.3.1 just went live on the asset store, check out the release notes for all the details. This version includes magic leap support! Thankyou everyone here for your help with testing this. Please consider giving us a review on the asset store if you're happy with how this went :)

Placeholder-Software / Dissonance

[feature] Magic Leap Support #125