Building for Quest 2 (Unity, Android NDK Bionic build)

[Vivraan]

I'm currently using Concentus.Native 1.5.2 with the latest Concentus package in Unity, and I'm building for Quest 2 (possibly 3 later).

The C# implementation should work out of the box but I would prefer native bindings if that's available as the app must update audio in realtime.

I think the code should take care of selecting the right native encoder, but would I need to build for Quest 2 separately using JNI? Based on how Unity uses IL2CPP, what kind of behaviour can I expect?

[lostromb]

I have no idea. I was hoping you could tell me. Based on my cursory knowledge of Unity, older versions would compile the game code into /Managed/Assembly-CSharp.dll, and any additional C# dependencies would go in there and more or less run as a normal .Net program. With this setup you could maybe dump the native opus binaries in there and they would just work. With il2cpp, the only place to put native dependencies seems to be using Plugins. Based on that documentation it seems plausible that putting the opus-arm64 native binaries as a project plugin for the Quest 2 should let the C# code just pick them up with its existing [DllImport] and no changes required.

[Vivraan]

Well, I managed to build the project; now, all I need to see is the logs. Besides, internally Android builds include the included libopus.so.

[Vivraan]

The implementation is unable to use the native libraries. I have the logging set up correctly, but it would have been great to use the native libs here.

Here, I did not explicitly place libopus.so in Plugins/, I'm trying that now.

[lostromb]

Besides, internally Android builds include the included libopus.so.

My limited experience with native libs on Android is that if you're trying to reference a built-in system library like /usr/lib/libopus.so, for some reason C# cannot find that library using [DllImport]. Not sure if it's an Android app sandboxing thing or just a quirk with the C# runtime. I have been able to load native libs if you explicitly bundle your own copy of the library into the .apk. Not sure about where exactly in the apk to put it or what to name it. You might have to just play around. Copy the Concentus CSharp/native/* source code into your project and add logs or breakpoints to try and see what it's doing within NativePlatformUtils.PrepareNativeLibrary

[Vivraan]

Okay so here are the logs:

 Server (0): Preparing native library "opus"
                   at Concentus.Native.NativePlatformUtils.PrepareNativeLibrary (System.String libraryName, System.IO.TextWriter logger) <0x00000 + 0xffffffff> 0 in <00000000000000000000000000000000>:0 
                   at Concentus.Native.NativeOpus.Initialize (System.IO.TextWriter logger) <0x00000 + 0xffffffff> 0 in <00000000000000000000000000000000>:0 
                   at Concentus.OpusCodecFactory.NativeLibraryAvailable (System.IO.TextWriter messageLogger) <0x00000 + 0xffffffff> 0 in <00000000000000000000000000000000>:0 
                   at Concentus.OpusCodecFactory.CreateEncoder (System.Int32 sampleRate, System.Int32 numChannels, Concentus.Enums.OpusApplication application, System.IO.TextWriter messageLogger) <0x00000 + 0xffffffff> 0 in <00000000000000000000000000000000>:0 
                   at MultiCollabSys.Runtime.FirebaseAudioUploader+<>c__DisplayClass15_0.<OnAudioFilterRead>g__FirebaseStorageConsumerAsync|0 () <0x00000 + 0xffffffff> 0 in <00000000000000000000000000000000>:0 
                   at System.Runtime.CompilerServices.AsyncTaskMethodBui
08:14:33.004  I  Server (0): Detected current platform as "unknown-arm64"
                   at Concentus.Native.NativePlatformUtils.PrepareNativeLibrary (System.String libraryName, System.IO.TextWriter logger) <0x00000 + 0xffffffff> 0 in <00000000000000000000000000000000>:0 
                   at Concentus.Native.NativeOpus.Initialize (System.IO.TextWriter logger) <0x00000 + 0xffffffff> 0 in <00000000000000000000000000000000>:0 
                   at Concentus.OpusCodecFactory.NativeLibraryAvailable (System.IO.TextWriter messageLogger) <0x00000 + 0xffffffff> 0 in <00000000000000000000000000000000>:0 
                   at Concentus.OpusCodecFactory.CreateEncoder (System.Int32 sampleRate, System.Int32 numChannels, Concentus.Enums.OpusApplication application, System.IO.TextWriter messageLogger) <0x00000 + 0xffffffff> 0 in <00000000000000000000000000000000>:0 
                   at MultiCollabSys.Runtime.FirebaseAudioUploader+<>c__DisplayClass15_0.<OnAudioFilterRead>g__FirebaseStorageConsumerAsync|0 () <0x00000 + 0xffffffff> 0 in <00000000000000000000000000000000>:0 
                   at System.Runtime.CompilerServices.Async
08:14:33.006  I  Server (0): Probing for an already existing opus
                   at Concentus.Native.NativePlatformUtils.PrepareNativeLibrary (System.String libraryName, System.IO.TextWriter logger) <0x00000 + 0xffffffff> 0 in <00000000000000000000000000000000>:0 
                   at Concentus.Native.NativeOpus.Initialize (System.IO.TextWriter logger) <0x00000 + 0xffffffff> 0 in <00000000000000000000000000000000>:0 
                   at Concentus.OpusCodecFactory.NativeLibraryAvailable (System.IO.TextWriter messageLogger) <0x00000 + 0xffffffff> 0 in <00000000000000000000000000000000>:0 
                   at Concentus.OpusCodecFactory.CreateEncoder (System.Int32 sampleRate, System.Int32 numChannels, Concentus.Enums.OpusApplication application, System.IO.TextWriter messageLogger) <0x00000 + 0xffffffff> 0 in <00000000000000000000000000000000>:0 
                   at MultiCollabSys.Runtime.FirebaseAudioUploader+<>c__DisplayClass15_0.<OnAudioFilterRead>g__FirebaseStorageConsumerAsync|0 () <0x00000 + 0xffffffff> 0 in <00000000000000000000000000000000>:0 
                   at System.Runtime.CompilerServices.AsyncTaskMeth
08:14:33.007  I  Server (0): Failed to resolve native library "opus".
                   at Concentus.Native.NativePlatformUtils.PrepareNativeLibrary (System.String libraryName, System.IO.TextWriter logger) <0x00000 + 0xffffffff> 0 in <00000000000000000000000000000000>:0 
                   at Concentus.Native.NativeOpus.Initialize (System.IO.TextWriter logger) <0x00000 + 0xffffffff> 0 in <00000000000000000000000000000000>:0 
                   at Concentus.OpusCodecFactory.NativeLibraryAvailable (System.IO.TextWriter messageLogger) <0x00000 + 0xffffffff> 0 in <00000000000000000000000000000000>:0 
                   at Concentus.OpusCodecFactory.CreateEncoder (System.Int32 sampleRate, System.Int32 numChannels, Concentus.Enums.OpusApplication application, System.IO.TextWriter messageLogger) <0x00000 + 0xffffffff> 0 in <00000000000000000000000000000000>:0 
                   at MultiCollabSys.Runtime.FirebaseAudioUploader+<>c__DisplayClass15_0.<OnAudioFilterRead>g__FirebaseStorageConsumerAsync|0 () <0x00000 + 0xffffffff> 0 in <00000000000000000000000000000000>:0 
                   at System.Runtime.CompilerServices.AsyncTask
08:14:33.009  I  Server (0): Is native opus available? False
                   at Concentus.OpusCodecFactory.NativeLibraryAvailable (System.IO.TextWriter messageLogger) <0x00000 + 0xffffffff> 0 in <00000000000000000000000000000000>:0 
                   at Concentus.OpusCodecFactory.CreateEncoder (System.Int32 sampleRate, System.Int32 numChannels, Concentus.Enums.OpusApplication application, System.IO.TextWriter messageLogger) <0x00000 + 0xffffffff> 0 in <00000000000000000000000000000000>:0 
                   at MultiCollabSys.Runtime.FirebaseAudioUploader+<>c__DisplayClass15_0.<OnAudioFilterRead>g__FirebaseStorageConsumerAsync|0 () <0x00000 + 0xffffffff> 0 in <00000000000000000000000000000000>:0 
                   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[TStateMachine] (TStateMachine& stateMachine) <0x00000 + 0xffffffff> 0 in <00000000000000000000000000000000>:0 
                   at MultiCollabSys.Runtime.FirebaseAudioUploader+<>c__DisplayClass15_0.<OnAudioFilterRead>g__FirebaseStorageConsumerAsync|0 () <0x00000 + 0xffffffff> 0 in <00000000000000000000000000000000>:0 
                   at Syste
08:14:33.024  I  Server (0): Got managed Opus encoder, Concentus 2.1.2.
                   at MultiCollabSys.Runtime.FirebaseAudioUploader+<>c__DisplayClass15_0.<OnAudioFilterRead>g__FirebaseStorageConsumerAsync|0 () <0x00000 + 0xffffffff> 0 in <00000000000000000000000000000000>:0 
                   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[TStateMachine] (TStateMachine& stateMachine) <0x00000 + 0xffffffff> 0 in <00000000000000000000000000000000>:0 
                   at MultiCollabSys.Runtime.FirebaseAudioUploader+<>c__DisplayClass15_0.<OnAudioFilterRead>g__FirebaseStorageConsumerAsync|0 () <0x00000 + 0xffffffff> 0 in <00000000000000000000000000000000>:0 
                   at System.Threading.Tasks.Task`1[TResult].InnerInvoke () <0x00000 + 0xffffffff> 0 in <00000000000000000000000000000000>:0 
                   at System.Threading.Tasks.Task.Execute () <0x00000 + 0xffffffff> 0 in <00000000000000000000000000000000>:0

It looks like it's not finding the correct library.

It seems to be skipping over the check for Android, and that's just from what's decompiled.

In your implementation of GetCurrentPlatformInternal, the case for Android doesn't seem to be handled. I guess the reason is fairly obvious -- there isn't a way to do this in any kind of .NET without relying on some framework that specifically compiles C# code to Android or iOS. https://github.com/lostromb/concentus/blob/e42b40f757feea1df9bcb336e66d70e792d2790c/CSharp/Concentus/Native/NativePlatformUtils.cs#L69-L102

Note: I am on .NET Standard 2.1.

So far, I have just been reading the logs, so I'd need to get the source code or a mapping file to set breakpoints.

[Vivraan]

My current recourse is to copy the Concentus C# codebase into my project and add a Unity-specific runtime check for Android for my case.

It's strange that Linux is not selected here at the minimum.

[Vivraan]

Here's my fork: https://github.com/Vivraan/concentus-unity. Note that to use it in Unity, I have to remove a bunch of the C# project files and assembly info files, but I haven't done so in the fork. Technically, I'd want to make it follow the UPM package specification (https://docs.unity3d.com/Manual/CustomPackages.html) for smooth interop, but I don't know for sure how to maintain a fork with significant changes from the base! 😅

Unity-specific change: https://github.com/Vivraan/concentus-unity/blob/9154952f58bdac731d1637cdabe642564f878d32/CSharp/Concentus/Native/NativePlatformUtils.cs#L93-L110

NB(vivraan): method for maintaining as UPM package

• Fork repo
• git init {Concentus.CSharp}
• use as submodule within UPM Package structure

[Vivraan]

So now we have:

15:49:37.622  D  Failed to load native plugin: Unable to load library '/data/app/~~BA1a_Z4_iFl3PQTO4UxZIA==/uk.ac.kingston.masterphil-2-vv7bA051uPwRVJQ92M6Q==/lib/arm64/libopus.so', error 'java.lang.UnsatisfiedLinkError: dlopen failed: library "libm.so.6" not found: needed by /data/app/~~BA1a_Z4_iFl3PQTO4UxZIA==/uk.ac.kingston.masterphil-2-vv7bA051uPwRVJQ92M6Q==/lib/arm64/libopus.so in namespace classloader-namespace'
15:51:12.774  I  Server (0): Preparing native library "opus"
                   at MultiCollabSys.Runtime.UILogHandler.WriteLine (System.String format, System.Object arg0) [0x00024] in O:\DevKU\DevPhD\MasterPhil\Assets\MultiCollabSys\Runtime\UILogHandler.cs:93 
                   at Concentus.Native.NativePlatformUtils.PrepareNativeLibrary (System.String libraryName, System.IO.TextWriter logger) [0x0000d] in O:\DevKU\DevPhD\MasterPhil\Assets\Concentus\Native\NativePlatformUtils.cs:186 
                   at Concentus.Native.NativeOpus.Initialize (System.IO.TextWriter logger) [0x00007] in O:\DevKU\DevPhD\MasterPhil\Assets\Concentus\Native\NativeOpus.cs:15 
                   at Concentus.OpusCodecFactory.NativeLibraryAvailable (System.IO.TextWriter messageLogger) [0x00022] in O:\DevKU\DevPhD\MasterPhil\Assets\Concentus\OpusCodecFactory.cs:197 
                   at Concentus.OpusCodecFactory.CreateEncoder (System.Int32 sampleRate, System.Int32 numChannels, Concentus.Enums.OpusApplication application, System.IO.TextWriter messageLogger) [0x00009] in O:\DevKU\DevPhD\MasterPhil\Assets\Concentus\OpusCodecFactory
15:51:17.380  I  Server (0): Detected current platform as "android-arm64"
                   at MultiCollabSys.Runtime.UILogHandler.WriteLine (System.String format, System.Object arg0) [0x00024] in O:\DevKU\DevPhD\MasterPhil\Assets\MultiCollabSys\Runtime\UILogHandler.cs:93 
                   at Concentus.Native.NativePlatformUtils.PrepareNativeLibrary (System.String libraryName, System.IO.TextWriter logger) [0x0002b] in O:\DevKU\DevPhD\MasterPhil\Assets\Concentus\Native\NativePlatformUtils.cs:189 
                   at Concentus.Native.NativeOpus.Initialize (System.IO.TextWriter logger) [0x00007] in O:\DevKU\DevPhD\MasterPhil\Assets\Concentus\Native\NativeOpus.cs:15 
                   at Concentus.OpusCodecFactory.NativeLibraryAvailable (System.IO.TextWriter messageLogger) [0x00022] in O:\DevKU\DevPhD\MasterPhil\Assets\Concentus\OpusCodecFactory.cs:197 
                   at Concentus.OpusCodecFactory.CreateEncoder (System.Int32 sampleRate, System.Int32 numChannels, Concentus.Enums.OpusApplication application, System.IO.TextWriter messageLogger) [0x00009] in O:\DevKU\DevPhD\MasterPhil\Assets\Concentus\Opu
15:52:14.513  I  Server (0): Probing for libopus.so within local Android .apk
                   at MultiCollabSys.Runtime.UILogHandler.WriteLine (System.String value) [0x0001f] in O:\DevKU\DevPhD\MasterPhil\Assets\MultiCollabSys\Runtime\UILogHandler.cs:108 
                   at Concentus.Native.NativePlatformUtils.PrepareNativeLibrary (System.String libraryName, System.IO.TextWriter logger) [0x000a1] in O:\DevKU\DevPhD\MasterPhil\Assets\Concentus\Native\NativePlatformUtils.cs:205 
                   at Concentus.Native.NativeOpus.Initialize (System.IO.TextWriter logger) [0x00007] in O:\DevKU\DevPhD\MasterPhil\Assets\Concentus\Native\NativeOpus.cs:15 
                   at Concentus.OpusCodecFactory.NativeLibraryAvailable (System.IO.TextWriter messageLogger) [0x00022] in O:\DevKU\DevPhD\MasterPhil\Assets\Concentus\OpusCodecFactory.cs:197 
                   at Concentus.OpusCodecFactory.CreateEncoder (System.Int32 sampleRate, System.Int32 numChannels, Concentus.Enums.OpusApplication application, System.IO.TextWriter messageLogger) [0x00009] in O:\DevKU\DevPhD\MasterPhil\Assets\Concentus\OpusCodecFactory.cs
15:52:19.891  I  Server (0): Attempting to load libopus.so as a linux .so
                   at MultiCollabSys.Runtime.UILogHandler.WriteLine (System.String value) [0x0001f] in O:\DevKU\DevPhD\MasterPhil\Assets\MultiCollabSys\Runtime\UILogHandler.cs:108 
                   at Concentus.Native.NativePlatformUtils.ProbeLibrary (System.String libName, Concentus.Native.OSAndArchitecture platformInfo, System.IO.TextWriter logger) [0x001d1] in O:\DevKU\DevPhD\MasterPhil\Assets\Concentus\Native\NativePlatformUtils.cs:879 
                   at Concentus.Native.NativePlatformUtils.PrepareNativeLibrary (System.String libraryName, System.IO.TextWriter logger) [0x000aa] in O:\DevKU\DevPhD\MasterPhil\Assets\Concentus\Native\NativePlatformUtils.cs:206 
                   at Concentus.Native.NativeOpus.Initialize (System.IO.TextWriter logger) [0x00007] in O:\DevKU\DevPhD\MasterPhil\Assets\Concentus\Native\NativeOpus.cs:15 
                   at Concentus.OpusCodecFactory.NativeLibraryAvailable (System.IO.TextWriter messageLogger) [0x00022] in O:\DevKU\DevPhD\MasterPhil\Assets\Concentus\OpusCodecFactory.cs:197 
                   at Conce
15:52:19.915  I  Server (0): Error while loading library libopus.so: dlopen failed: library "libm.so.6" not found: needed by /data/app/~~BA1a_Z4_iFl3PQTO4UxZIA==/uk.ac.kingston.masterphil-2-vv7bA051uPwRVJQ92M6Q==/lib/arm64/libopus.so in namespace classloader-namespace
                   at MultiCollabSys.Runtime.UILogHandler.WriteLine (System.String value) [0x0001f] in O:\DevKU\DevPhD\MasterPhil\Assets\MultiCollabSys\Runtime\UILogHandler.cs:108 
                   at Concentus.Native.NativePlatformUtils.ProbeLibrary (System.String libName, Concentus.Native.OSAndArchitecture platformInfo, System.IO.TextWriter logger) [0x0023d] in O:\DevKU\DevPhD\MasterPhil\Assets\Concentus\Native\NativePlatformUtils.cs:889 
                   at Concentus.Native.NativePlatformUtils.PrepareNativeLibrary (System.String libraryName, System.IO.TextWriter logger) [0x000aa] in O:\DevKU\DevPhD\MasterPhil\Assets\Concentus\Native\NativePlatformUtils.cs:206 
                   at Concentus.Native.NativeOpus.Initialize (System.IO.TextWriter logger) [0x00007] in O:\DevKU\DevPhD\MasterPhil\Assets\Concentus\Native\NativeO
15:52:38.276  I  Server (0): Native library "opus" was not found in the local .apk
                   at MultiCollabSys.Runtime.UILogHandler.WriteLine (System.String format, System.Object arg0) [0x00024] in O:\DevKU\DevPhD\MasterPhil\Assets\MultiCollabSys\Runtime\UILogHandler.cs:93 
                   at Concentus.Native.NativePlatformUtils.PrepareNativeLibrary (System.String libraryName, System.IO.TextWriter logger) [0x000cc] in O:\DevKU\DevPhD\MasterPhil\Assets\Concentus\Native\NativePlatformUtils.cs:209 
                   at Concentus.Native.NativeOpus.Initialize (System.IO.TextWriter logger) [0x00007] in O:\DevKU\DevPhD\MasterPhil\Assets\Concentus\Native\NativeOpus.cs:15 
                   at Concentus.OpusCodecFactory.NativeLibraryAvailable (System.IO.TextWriter messageLogger) [0x00022] in O:\DevKU\DevPhD\MasterPhil\Assets\Concentus\OpusCodecFactory.cs:197 
                   at Concentus.OpusCodecFactory.CreateEncoder (System.Int32 sampleRate, System.Int32 numChannels, Concentus.Enums.OpusApplication application, System.IO.TextWriter messageLogger) [0x00009] in O:\DevKU\DevPhD\MasterPhil\Assets\Conc
15:52:43.709  I  Server (0): Is native opus available? False
                   at MultiCollabSys.Runtime.UILogHandler.WriteLine (System.String value) [0x0001f] in O:\DevKU\DevPhD\MasterPhil\Assets\MultiCollabSys\Runtime\UILogHandler.cs:108 
                   at Concentus.OpusCodecFactory.NativeLibraryAvailable (System.IO.TextWriter messageLogger) [0x00046] in O:\DevKU\DevPhD\MasterPhil\Assets\Concentus\OpusCodecFactory.cs:198 
                   at Concentus.OpusCodecFactory.CreateEncoder (System.Int32 sampleRate, System.Int32 numChannels, Concentus.Enums.OpusApplication application, System.IO.TextWriter messageLogger) [0x00009] in O:\DevKU\DevPhD\MasterPhil\Assets\Concentus\OpusCodecFactory.cs:74 
                   at MultiCollabSys.Runtime.FirebaseAudioUploader+<>c__DisplayClass15_0.<OnAudioFilterRead>g__FirebaseStorageConsumerAsync|0 () [0x00098] in O:\DevKU\DevPhD\MasterPhil\Assets\MultiCollabSys\Runtime\FirebaseAudioUploader.cs:177 
                   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[TStateMachine] (TStateMachine& stateMachine) <0x00000 + 0xffffffff> 0 in <000000000000
15:52:43.733  I  Server (0): Got managed Opus encoder, Concentus 2.1.2-debug-nonparity.
                   at UnityEngine.Logger.Log (UnityEngine.LogType logType, System.Object message) [0x00028] in \home\bokken\build\output\unity\unity\Runtime\Export\Logging\Logger.cs:60 
                   at UnityEngine.Debug.Log (System.Object message) [0x00008] in \home\bokken\build\output\unity\unity\Runtime\Export\Debug\Debug.bindings.cs:102 
                   at MultiCollabSys.Runtime.FirebaseAudioUploader+<>c__DisplayClass15_0.<OnAudioFilterRead>g__FirebaseStorageConsumerAsync|0 () [0x0012c] in O:\DevKU\DevPhD\MasterPhil\Assets\MultiCollabSys\Runtime\FirebaseAudioUploader.cs:185 
                   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[TStateMachine] (TStateMachine& stateMachine) <0x00000 + 0xffffffff> 0 in <00000000000000000000000000000000>:0 
                   at MultiCollabSys.Runtime.FirebaseAudioUploader+<>c__DisplayClass15_0.<OnAudioFilterRead>g__FirebaseStorageConsumerAsync|0 () <0x00000 + 0xffffffff> 0 in <00000000000000000000000000000000>:0 
                   at System.Threading.Tasks.Task`1[TR

EDIT: Library exists in the correct location:

Reference: https://github.com/lostromb/concentus/blob/e42b40f757feea1df9bcb336e66d70e792d2790c/CSharp/Concentus/Native/NativePlatformUtils.cs#L192-L205 You did mention here that dlopen would fail if a shared system location couldn't be accessed, but here the correct .so library is located and dlopen fails because there's another dependency (libm.so.6)

[lostromb]

Yeah, you've got it mostly correct. Problem 1 is that the code doesn't know what operating system it's on because there's not a reliable way to detect Android OS using only .NetStandard 2.0 calls, which you have worked around. Problem 2 is that it can't actually load libopus because it depends on the linux common math library which doesn't load.

The android libopus.so may be built incorrectly (missing SONAME). I found some guidance here and here which may help.

[Vivraan]

Btw which version of libopus have you packaged in this project?

[Vivraan]

PS O:\DevKU\DevPhD\MasterPhil\Assets\Plugins\Android> readelf --dynamic .\libopus.so | rg NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

This would mean the given libopus.so won't run on Android. I tried getting libm.so.6 from the Debian repo, but I guess it would need Android ABI compatibility:

Plugin 'Assets/Plugins/Android/libs/arm64-v8a/libm.so.6' is not supported on Android, please deselect it in Plugin Inspector
System.Linq.Enumerable/WhereEnumerableIterator`1<object>:MoveNext ()
Bee.BeeDriver.BeeDriver:WriteDataForBuildProgram (Bee.BeeDriver.InternalState,System.Threading.Tasks.TaskCompletionSource`1<int>)
Bee.BeeDriver.BeeDriver/<>c__DisplayClass0_0:<BuildAsync>g__WriteData|0 ()
Bee.BeeDriver.BeeDriver:BuildAsync (Bee.BeeDriver.BuildRequest,System.Threading.CancellationToken)
UnityEngine.GUIUtility:ProcessEvent (int,intptr,bool&)

Is it possible to compile libopus.so from scratch for Android/Quest 2? The answer seems to be yes given this project exists: https://github.com/TyounanMOTI/UnityOpus/tree/master/Assets/Plugins/UnityOpus/Plugins/Android

There's also https://github.com/theeasiestway/android-opus-codec which is a codec implementation specifically for Android.

[lostromb]

If you trust random loose binaries in zip file attachments, then here is some version of libopus.so which I just got from adb pull /system/lib64/libopus.so from my phone. Presumably it is compiled correctly to run on android, maybe it'll behave properly for you.

If opus is already preinstalled into the /system/lib64 on the Quest as well maybe you should try and see if there's a way to load that instead of bundling a separate copy yourself.

update: I tried that and it didn't work

[linker] library "/system/lib64/libopus.so" ("/system/lib64/libopus.so") needed or dlopened by "/data/app/Durandal.AndroidClient-R4DhWb7V9HpvQQHkgs6ZdQ==/lib/arm64/libmonosgen-2.0.so" is not accessible for the namespace: [name="classloader-namespace", ld_library_paths="", default_library_paths="/data/app/Durandal.AndroidClient-R4DhWb7V9HpvQQHkgs6ZdQ==/lib/arm64:/data/app/Durandal.AndroidClient-R4DhWb7V9HpvQQHkgs6ZdQ==/base.apk!/lib/arm64-v8a", permitted_paths="/data:/mnt/expand:/data/data/Durandal.AndroidClient"]

libopus.so.zip

[Vivraan]

If you trust random loose binaries in zip file attachments, then here is some version of libopus.so which I just got from adb pull /system/lib64/libopus.so from my phone. Presumably it is compiled correctly to run on android, maybe it'll behave properly for you.

If opus is already preinstalled into the /system/lib64 on the Quest as well maybe you should try and see if there's a way to load that instead of bundling a separate copy yourself.

update: I tried that and it didn't work

[linker] library "/system/lib64/libopus.so" ("/system/lib64/libopus.so") needed or dlopened by "/data/app/Durandal.AndroidClient-R4DhWb7V9HpvQQHkgs6ZdQ==/lib/arm64/libmonosgen-2.0.so" is not accessible for the namespace: [name="classloader-namespace", ld_library_paths="", default_library_paths="/data/app/Durandal.AndroidClient-R4DhWb7V9HpvQQHkgs6ZdQ==/lib/arm64:/data/app/Durandal.AndroidClient-R4DhWb7V9HpvQQHkgs6ZdQ==/base.apk!/lib/arm64-v8a", permitted_paths="/data:/mnt/expand:/data/data/Durandal.AndroidClient"]

libopus.so.zip

No, for this specific reason, I am looking for either Opus 1.1.2 (as it would be from your repo's hardening of the API) or 1.3.1 (the version internally used by Unity Vivox).

There's loads of info linked on this thread (to my own posts :x) https://forum.unity.com/threads/how-to-simulate-laggy-lossy-communication-using-vivox-service.1593219/

[Vivraan]

Would you say there could be inconsistencies if I built Opus 1.3.1 and passed that to Concentus.Native?

Not that I don't trust your zip (lol), I'm just concerned about proper versioning.

[Vivraan]

    I'm encountering something really maddening.

    I attempted to build `libopus.so` from two sources:
    1. `git clone --branch v1.1.2 --single-branch https://github.com/xiph/opus.git`
    2. `wget http://downloads.xiph.org/releases/opus/opus-1.1.2.tar.gz`

    using the Android NDK supplied by Unity through WSL. The correct `clang(++)` executables for `aarch64-linux-android` are in the relevant toolchain subdirectory (strangely, only till API 31, even though Unity expressly permits API 32+ atm).

    EDIT: verified that it's Clang 12.0.8 targeting `aarch64-unknown-linux-android31`.

    Naturally (1) needs the following steps:
    ```bash
    $ ./autogen.sh
    $ ./configure --host=aarch64-linux-android --prefix=/home/vivraan/build --enable-shared --disable-static CC=$ANDROID_NDK_TOOLCHAIN/bin/aarch64-linux-android31-clang CXX=$ANDROID_NDK_TOOLCHAIN/bin/aarch64-linux-android31-clang++ | grep -e "static libraries" -e "shared libraries"
    $ make -j$(nproc); make install
    ```

    For (2), we skip `./autogen.sh`. I added `grep` to check whether I can get shared libraries.
    (In (2), `make check` doesn't work since the tests must be rebuilt.)

    Now, the maddening part:

    For (1), the included `libtool` (2.4.2) says shared libraries _can_ be built, but the `.so` fails to build with this error in the end:
    ```ahk
    /test_unit_types
      CCLD     opus_demo
      CCLD     repacketizer_demo
      CCLD     tests/test_opus_api
      CCLD     tests/test_opus_encode
      CCLD     tests/test_opus_padding
                                      clang: error: no such file or directory: './.libs/libopus.so'
    clang: error: no such file or directory: './.libs/libopus.so'
    clang: error: no such file or directory: './.libs/libopus.so'
    clang: error: no such file or directory: './.libs/libopus.so'
    clang: error: no such file or directory: './.libs/libopus.so'
    make[2]: *** [Makefile:2128: repacketizer_demo] Error 1
                                                           make[2]: *** Waiting for unfinished jobs....
                                                                                                       make[2]: *** [Makefile:2122: opus_demo] Error 1
      make[2]: *** [Makefile:2140: tests/test_opus_api] Error 1
                                                               make[2]: *** [Makefile:2152: tests/test_opus_encode] Error 1
                                                                                                                           make[2]: *** [Makefile:2158: tests/test_opus_padding] Error 1
                                        make[2]: Leaving directory '/home/vivraan/opus'
    make[1]: *** [Makefile:2527: all-recursive] Error 1
    make[1]: Leaving directory '/home/vivraan/opus'
    make: *** [Makefile:1553: all] Error 2
    make  install-recursive
    make[1]: Entering directory '/home/vivraan/opus'
    make[2]: Entering directory '/home/vivraan/opus'
      CCLD     opus_demo
    clang: error: no such file or directory: './.libs/libopus.so'
    make[2]: *** [Makefile:2122: opus_demo] Error 1
    make[2]: Leaving directory '/home/vivraan/opus'
    make[1]: *** [Makefile:2527: install-recursive] Error 1
    make[1]: Leaving directory '/home/vivraan/opus'
    make: *** [Makefile:3077: install] Error 2
    ```
    (Why would I be expecting libraries in "./.lib"? I didn't ask for that!)

    For (2), there is no included `libtool`, so I tried copying in (1)'s `libtool`, and then even apt-got `libtool` 2.4.7. For the exact same toolchain, it says shared libraries _can not_ be built. It correctly builds static libraries after implicitly configuring for the same (verified through `grep`ing the output of `./configure`).

    I am at a bit of a loss since I must copy these steps for `glibc`.

EDIT: One remedy seems to be just relying on WSL's gcc instead of the NDK clang. TEMP-FIX: Use gcc-aarch64-linux-gnu.

[Vivraan]

glibc flat out doesn't build for NDK's clang. However, glibc/configure does check for aarc64-linux-android-* compilers.

I installed aarch64-linux-gnu-gcc and rebuilt Opus 1.1.2 with it. Now I have a new (possibly link helper) dependency:

vivraan@Vivraan:~/build$ readelf --dynamic lib/libopus.so | grep -e NEEDED -e SONAME
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [ld-linux-aarch64.so.1]
 0x000000000000000e (SONAME)             Library soname: [libopus.so]

[Vivraan]

libopus.so.zip

I checked your file, and coincidentally it doesn't have a version string:

vivraan@Vivraan:~$ ls
android-ndk  build  glibc  libopus.so  libopus.so.zip  opus  opus-1.1.2  opus-1.1.2.tar.gz
vivraan@Vivraan:~$ strings libopus.so | grep -E '[0-9]+\.[0-9]+\.[0-9]+'
vivraan@Vivraan:~$ strings libopus.so | grep -E '[0-9]+\.[0-9]+\.[0-9]+'
vivraan@Vivraan:~$ strings libopus.so | grep libopus
libopus.so
libopus unknown-fixed
vivraan@Vivraan:~$ strings /home/vivraan/build/lib/libopus.so | grep libopus
libopus.so
libopus 1.1.2
vivraan@Vivraan:~$

[lostromb]

Would you say there could be inconsistencies if I built Opus 1.3.1 and passed that to Concentus.Native?

Not that I don't trust your zip (lol), I'm just concerned about proper versioning.

The only libopus change I can think of which might affect your use case was that somewhere between 1.2.0 and 1.5.0 they added support for packet loss concealment above 40Kbps bitrates. Besides that it was mostly internal stuff for better quality at low bitrates, ambisonics support, some more fancy analysis and experimental new features that don't affect the public API.

I'm gonna try packing libm.so into my own apk and see if I can get things to load that way.

[Vivraan]

Would you say there could be inconsistencies if I built Opus 1.3.1 and passed that to Concentus.Native? Not that I don't trust your zip (lol), I'm just concerned about proper versioning.

The only libopus change I can think of which might affect your use case was that somewhere between 1.2.0 and 1.5.0 they added support for packet loss concealment above 40Kbps bitrates. Besides that it was mostly internal stuff for better quality at low bitrates, ambisonics support, some more fancy analysis and experimental new features that don't affect the public API.

I'm gonna try packing libm.so into my own apk and see if I can get things to load that way.

The second thing you mentioned is definitely an imperative for my application (better quality at low bitrates, implying there's a difference in audio perception).

1.5.2 at the head seems to be built for better deep learning analysis, which may have been useful for our case. The home page suggests afaik that it is fully backward compatible, so perhaps it might be a good choice to target that, but I want to know for sure if I can use Concentus as a wrapper in that case.

If you haven't already guessed this is the most important technical thread for me at this very moment. There can be no half-measures.

Please let me know if you have luck building glibc on aarch64-linux-android, as I want to replicate your steps.

[lostromb]

I have no idea how to build native libs for android. I was just gonna copy loose binaries that I already had and try and see if there's a way for C# to load them correctly, using a sandbox Xamarin app to work in.

Based on what I've read from your Vivox threads, and assuming the encoded audio is not actually used in real-time, it still seems like the simplest way to get precisely what you want in terms of quality and speed is by building libopus 1.3.1 for desktop (so you have parity with what Vivox uses), capture the samples from your unity app as a filter, and then pipe those raw samples to an external server which will then do the opus encode offline.

[lostromb]

Bingo! I managed to get it to work with the attached files. This is the /system/lib64/libm.so file from an android 8.0 phone, and a build a libopus 1.3.1 from https://github.com/theeasiestway/android-opus-codec. If you bundle both libs together in the apk then opus loads and decodes audio correctly (haven't tested encoding but I don't see why it wouldn't work)

arm64-v8a.zip

[Vivraan]

Just checked out your files, but unsure why I'm seeing what I'm seeing:

PS O:\DevKU\DevPhD\arm64-v8a> readelf --dynamic .\libopus.so | rg -e NEEDED -e SONAME
 0x0000000000000001 (NEEDED)             Shared library: [libm.so]
 0x0000000000000001 (NEEDED)             Shared library: [liblog.so]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so]
 0x000000000000000e (SONAME)             Library soname: [libopus.so]
PS O:\DevKU\DevPhD\arm64-v8a> readelf --dynamic .\libm.so | rg -e NEEDED -e SONAME
 0x0000000000000001 (NEEDED)             Shared library: [libc.so]
 0x000000000000000e (SONAME)             Library soname: [libm.so]

PS O:\DevKU\DevPhD\arm64-v8a> strings .\libm.so | rg [0-9]\.[0-9]+
5.5G
gold 1.11
PS O:\DevKU\DevPhD\arm64-v8a> strings .\libm.so | rg [0-9]\.[0-9]+
5.5G
gold 1.11
PS O:\DevKU\DevPhD\arm64-v8a> strings .\libopus.so | rg libopus
libopus.so
libopus unknown-fixed

Why does libopus have so many bizarre dependencies (libc.so is understandable as that's just a linking helper, maybe libstdc++, but liblog and libdl not so much!), and why is its version string not there... also I would've been expecting libm.so to be at 2.34+ at least.

It's a great start of course! Let's refine the process.

[Vivraan]

I have no idea how to build native libs for android. I was just gonna copy loose binaries that I already had and try and see if there's a way for C# to load them correctly, using a sandbox Xamarin app to work in.

Based on what I've read from your Vivox threads, and assuming the encoded audio is not actually used in real-time, it still seems like the simplest way to get precisely what you want in terms of quality and speed is by building libopus 1.3.1 for desktop (so you have parity with what Vivox uses), capture the samples from your unity app as a filter, and then pipe those raw samples to an external server which will then do the opus encode offline.

I don't think we can afford to push raw data online, so we are performing encoding locally on the Quest 2 headset. The audio output even from the managed encoder is okay, but I really want it to be as fast as possible.

Also, the re-encoded audio will definitely be used in realtime. We plan to be able to play them back within the application at some later stage, or better yet run a fast local model on the headset to generate latent info from the audio.

[Vivraan]

Just checked out your files, but unsure why I'm seeing what I'm seeing:

PS O:\DevKU\DevPhD\arm64-v8a> readelf --dynamic .\libopus.so | rg -e NEEDED -e SONAME
 0x0000000000000001 (NEEDED)             Shared library: [libm.so]
 0x0000000000000001 (NEEDED)             Shared library: [liblog.so]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so]
 0x000000000000000e (SONAME)             Library soname: [libopus.so]
PS O:\DevKU\DevPhD\arm64-v8a> readelf --dynamic .\libm.so | rg -e NEEDED -e SONAME
 0x0000000000000001 (NEEDED)             Shared library: [libc.so]
 0x000000000000000e (SONAME)             Library soname: [libm.so]

PS O:\DevKU\DevPhD\arm64-v8a> strings .\libm.so | rg [0-9]\.[0-9]+
5.5G
gold 1.11
PS O:\DevKU\DevPhD\arm64-v8a> strings .\libm.so | rg [0-9]\.[0-9]+
5.5G
gold 1.11
PS O:\DevKU\DevPhD\arm64-v8a> strings .\libopus.so | rg libopus
libopus.so
libopus unknown-fixed

Why does libopus have so many bizarre dependencies (libc.so is understandable as that's just a linking helper, maybe libstdc++, but liblog and libdl not so much!), and why is its version string not there... also I would've been expecting libm.so to be at 2.34+ at least.

It's a great start of course! Let's refine the process.

The answer might be in how libopus is compiled for Android using the Bionic libc and not glibc.

[Vivraan]

Just revised the build process for libopus for Android.

~/ndk-setup.sh

export NDK=~/android-ndk-r26d # LTS, download from official Android NDK website
export TOOLCHAIN=$NDK/toolchains/llvm/prebuilt/linux-x86_64
export AR=$TOOLCHAIN/bin/llvm-ar
export CC=$TOOLCHAIN/bin/$TARGET$API-clang
export AS=$CC
export CXX=$TOOLCHAIN/bin/$TARGET$API-clang++
export LD=$TOOLCHAIN/bin/ld.lld
export RANLIB=$TOOLCHAIN/bin/llvm-ranlib
export STRIP=$TOOLCHAIN/bin/llvm-strip

% mkdir -p ~/build/opus/bionic
% sh ~/ndk-setup.sh
% cd ~/build/opus/bionic
% ~/opus/configure --host=$TARGET --prefix=/home/$USER/build/opus/bionic --disable-static --enable-shared
% make clean && make -j$(nproc)
% readelf --dynamic .libs/libopus.so | grep -e NEEDED -e SONAME
% strings .libs/libopus.so | grep libopus

I got these:

vivraan@Vivraan:~/build/opus/bionic$ readelf --dynamic .libs/libopus.so | grep -e NEEDED -e SONAME
 0x0000000000000001 (NEEDED)             Shared library: [libm.so]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so]
 0x000000000000000e (SONAME)             Library soname: [libopus.so]
vivraan@Vivraan:~/build/opus/bionic$ strings .libs/libopus.so | grep libopus
libopus.so
libopus 1.5.2

Android NDK has the requisite shared libraries under $TOOLCHAIN/sysroot/usr/lib/$TARGET/$API.

This is awesome. Unity likes these files now.

[Vivraan]

14:19:31.166  I  Server (0): Is native opus available? True
                   at MultiCollabSys.Runtime.UILogHandler.WriteLine (System.String value) [0x0001f] in O:\DevKU\DevPhD\MasterPhil\Assets\MultiCollabSys\Runtime\UILogHandler.cs:108 
                   at Concentus.OpusCodecFactory.NativeLibraryAvailable (System.IO.TextWriter messageLogger) [0x00046] in O:\DevKU\DevPhD\MasterPhil\Assets\Concentus\OpusCodecFactory.cs:198 
                   at Concentus.OpusCodecFactory.CreateEncoder (System.Int32 sampleRate, System.Int32 numChannels, Concentus.Enums.OpusApplication application, System.IO.TextWriter messageLogger) [0x00009] in O:\DevKU\DevPhD\MasterPhil\Assets\Concentus\OpusCodecFactory.cs:74 
                   at MultiCollabSys.Runtime.FirebaseAudioUploader+<>c__DisplayClass15_0.<OnAudioFilterRead>g__FirebaseStorageConsumerAsync|0 () [0x000a9] in O:\DevKU\DevPhD\MasterPhil\Assets\MultiCollabSys\Runtime\FirebaseAudioUploader.cs:178 
                   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[TStateMachine] (TStateMachine& stateMachine) <0x00000 + 0xffffffff> 0 in <0000000000000
14:19:31.169  I  Server (0): Got native Opus encoder, libopus 1.5.2.
                   at UnityEngine.Logger.Log (UnityEngine.LogType logType, System.Object message) [0x00028] in \home\bokken\build\output\unity\unity\Runtime\Export\Logging\Logger.cs:60 
                   at UnityEngine.Debug.Log (System.Object message) [0x00008] in \home\bokken\build\output\unity\unity\Runtime\Export\Debug\Debug.bindings.cs:102 
                   at MultiCollabSys.Runtime.FirebaseAudioUploader+<>c__DisplayClass15_0.<OnAudioFilterRead>g__FirebaseStorageConsumerAsync|0 () [0x0013d] in O:\DevKU\DevPhD\MasterPhil\Assets\MultiCollabSys\Runtime\FirebaseAudioUploader.cs:186 
                   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[TStateMachine] (TStateMachine& stateMachine) <0x00000 + 0xffffffff> 0 in <00000000000000000000000000000000>:0 
                   at MultiCollabSys.Runtime.FirebaseAudioUploader+<>c__DisplayClass15_0.<OnAudioFilterRead>g__FirebaseStorageConsumerAsync|0 () <0x00000 + 0xffffffff> 0 in <00000000000000000000000000000000>:0 
                   at System.Threading.Tasks.Task`1[TResult].InnerInvoke 

@lostromb, you now have all the steps needed for this specific task! Thank you for providing this fantastic project; it wouldn't be possible without your efforts. I'm closing this thread for now.

[lostromb]

Would you mind sharing your final binary file? There might be a good way to distribute it in the Concentus.Native package for other Android users so it just works out of the box for them.

[Vivraan]

Opus.zip I included the shared libs from Android NDK r26 LTS, but those can be replaced. This is libopus.so 1.5.2 though, so I'd need to rebuild for other versions.

[Vivraan]

The thing is, even iOS is not handled in the plugin picking API, so someone with a Mac would have to chime in for that... I could cover the Unity bit anyhow.

[swordmaster2k]

I got this working in the Godot Game engine with C# and the Meta Quest. Great information here!