Expose JNI method signatures for Android

rtumelty commented 6 years ago

Hi, I've been trying to get Porcupine working from Unity and C# for Android devices. Using the System.Runtime.InteropServices library (for interacting with C and C++ code), I can load the Porcupine library, and was able to determine the JNI function names. I can successfully call functions with no parameters (getFrameLength, getSampleRate). However when I try to call the init function, Porcupine causes a segmentation fault.

I have tried multiple different parameter and return types, and methods of marshalling the types between C# and the shared library, with no success. It's mostly guess work however, as the library contains no debug symbols and I don't know how the functions are defined in the JNI - it is undocumented as far as I can see. I have also (in desperation) tried using the arm builds for linux and raspberry pi, but these seem to be incompatible with armv7 and arm64.

This is a major blocking issue on a product that otherwise Porcupine would be perfect for. There are a couple of things that you might be able to provide that could help:

Provide a debug version of the shared library, so I can better diagnose the issue.
Provide the method signatures for the JNI wrapper. I requested this before, but was told there was no plans to open source the JNI code as it contains proprietary information. However I don't need the full source, and external method signatures should not be sensitive information.
I saw another issue relating to the lack of native C linking in the Android library (https://github.com/Picovoice/Porcupine/issues/73) in which you say you were going to look into exposing the functions for release v1.5. Do you think this is still the case, and can you provide a date estimate on that release?
Alternatively, if you are aware of anyone who has successfully integrated Porcupine in Unity Android, I'd really appreciate if you could direct me to a means of contacting them!

If you can help with any of these, I'd be very grateful! I realise that this is not the intended use of the JNI wrapper, however short of writing a Java wrapper for the JNI wrapper and calling that, I'm stuck for solutions. I've included a stack trace and some sample C# below. Thanks!

Looking forward to your response, Ronan

Expected behavior

Call Porcupine init from Unity using System.Runtime.InteropServices. Porcupine initialises.

Actual behavior

Segmentation fault when init is called. Stack trace below.

2018-11-06 11:24:18.864 23059-23089/com.meowtek.commandandcontrol E/CRASH: pid: 23059, tid: 23089, name: UnityMain >>> com.meowtek.commandandcontrol <<< 2018-11-06 11:24:19.109 23059-23089/com.meowtek.commandandcontrol E/CRASH: #01 pc 00009c7c ( (wrapper managed-to-native) PorcupineManager:init (string,string[],single[]) + 0x3cc (0xdcc548b0 0xdcc54f24) [0xede2dee0 - Unity Root Domain]+40060) 2018-11-06 11:24:19.109 23059-23089/com.meowtek.commandandcontrol E/CRASH: #03 il 00000394 at PorcupineManager.Init () [0x0009b] in /Users/Ronan/Unity Projects/CommandAndControl/Assets/Scripts/Porcupine/PorcupineManager.cs:45 2018-11-06 11:24:19.109 23059-23089/com.meowtek.commandandcontrol E/CRASH: #04 il 00000287 at PorcupineTest/c__Iterator0.MoveNext () [0x0005d] in /Users/Ronan/Unity Projects/CommandAndControl/Assets/Scripts/Porcupine/PorcupineTest.cs:17 2018-11-06 11:24:19.109 23059-23089/com.meowtek.commandandcontrol E/CRASH: #05 il 000001a3 at UnityEngine.SetupCoroutine.InvokeMoveNext (System.Collections.IEnumerator,intptr) [0x00028] in /Users/builduser/buildslave/unity/build/Runtime/Export/Coroutines.cs:17 2018-11-06 11:24:19.113 23059-23089/com.meowtek.commandandcontrol E/CRASH: #02 pc 00fbc8a4 /data/app/com.meowtek.commandandcontrol-0eFMZbq9Okv-j8w4eNgA0A==/lib/arm/libunity.so (ScriptingInvocation::Invoke(ScriptingExceptionPtr, bool)+84) 2018-11-06 11:24:19.113 23059-23089/com.meowtek.commandandcontrol E/CRASH: #03 pc 00dc8388 /data/app/com.meowtek.commandandcontrol-0eFMZbq9Okv-j8w4eNgA0A==/lib/arm/libunity.so (Coroutine::InvokeMoveNext(ScriptingExceptionPtr)+164) 2018-11-06 11:24:19.113 23059-23089/com.meowtek.commandandcontrol E/CRASH: #04 pc 00dc7ef8 /data/app/com.meowtek.commandandcontrol-0eFMZbq9Okv-j8w4eNgA0A==/lib/arm/libunity.so (Coroutine::Run(bool)+44) 2018-11-06 11:24:19.113 23059-23089/com.meowtek.commandandcontrol E/CRASH: #05 pc 00a309a4 /data/app/com.meowtek.commandandcontrol-0eFMZbq9Okv-j8w4eNgA0A==/lib/arm/libunity.so (DelayedCallManager::Update(int)+500) 2018-11-06 11:24:19.113 23059-23089/com.meowtek.commandandcontrol E/CRASH: #06 pc 00d9f624 /data/app/com.meowtek.commandandcontrol-0eFMZbq9Okv-j8w4eNgA0A==/lib/arm/libunity.so (InitPlayerLoopCallbacks()::UpdateScriptRunDelayedDynamicFrameRateRegistrator::Forward()+60) 2018-11-06 11:24:19.113 23059-23089/com.meowtek.commandandcontrol E/CRASH: #07 pc 00da01d4 /data/app/com.meowtek.commandandcontrol-0eFMZbq9Okv-j8w4eNgA0A==/lib/arm/libunity.so (ExecutePlayerLoop(NativePlayerLoopSystem)+96) 2018-11-06 11:24:19.113 23059-23089/com.meowtek.commandandcontrol E/CRASH: #08 pc 00da0224 /data/app/com.meowtek.commandandcontrol-0eFMZbq9Okv-j8w4eNgA0A==/lib/arm/libunity.so (ExecutePlayerLoop(NativePlayerLoopSystem)+176) 2018-11-06 11:24:19.113 23059-23089/com.meowtek.commandandcontrol E/CRASH: #09 pc 00da03cc /data/app/com.meowtek.commandandcontrol-0eFMZbq9Okv-j8w4eNgA0A==/lib/arm/libunity.so (PlayerLoop()+284) 2018-11-06 11:24:19.113 23059-23089/com.meowtek.commandandcontrol E/CRASH: #10 pc 0061d470 /data/app/com.meowtek.commandandcontrol-0eFMZbq9Okv-j8w4eNgA0A==/lib/arm/libunity.so (UnityPlayerLoop()+572) 2018-11-06 11:24:19.113 23059-23089/com.meowtek.commandandcontrol E/CRASH: #11 pc 00620bb4 /data/app/com.meowtek.commandandcontrol-0eFMZbq9Okv-j8w4eNgA0A==/lib/arm/libunity.so (nativeRender(_JNIEnv, _jobject*)+52) 2018-11-06 11:24:19.117 23059-23089/com.meowtek.commandandcontrol E/MessageQueue: IdleHandler threw exception java.lang.Error: signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 636475d3 Build fingerprint: 'samsung/starltexx/starlte:8.0.0/R16NW/G960FXXU2BRI3:user/release-keys' Revision: '26' pid: 23059, tid: 23089, name: UnityMain >>> com.meowtek.commandandcontrol <<< r0 6364732f r1 ca4f70b8 r2 e39ebcd0 r3 c29cdc1d r4 c1d3ace8 r5 ca4f7564 r6 00000000 r7 ca4f70b0 r8 00000000 r9 c0be26c0 sl c3cb684c fp c29cdc1d ip ca4f7108 sp ca4f7080 lr dcc54c80 pc c29cdc50 cpsr ee321c90
    at libpv_porcupine.Java_ai_picovoice_porcupine_Porcupine_init(Java_ai_picovoice_porcupine_Porcupine_init:51)
    at Unknown.00009c7c(Unknown Source:0)
    at PorcupineManager.init(Native Method)
    at PorcupineManager.Init(PorcupineManager.cs:45)
    at <Start>c__Iterator0.MoveNext(PorcupineTest.cs:17)
    at SetupCoroutine.InvokeMoveNext(Coroutines.cs:17)
    at <Module>.runtime_invoke_void_object_intptr(Native Method)
    at libmono.000225d3(Native Method)
    at libmono.mono_runtime_invoke(mono_runtime_invoke:136)
    at libunity.ScriptingInvocation::Invoke(ScriptingExceptionPtr*, bool)(Invoke:84)
    at libunity.Coroutine::InvokeMoveNext(ScriptingExceptionPtr*)(InvokeMoveNext:164)
    at libunity.Coroutine::Run(bool*)(Run:44)
    at libunity.DelayedCallManager::Update(int)(Update:500)
    at libunity.InitPlayerLoopCallbacks()::UpdateScriptRunDelayedDynamicFrameRateRegistrator::Forward()(InitPlayerLoopCallbacks:60)
    at libunity.ExecutePlayerLoop(NativePlayerLoopSystem*)(ExecutePlayerLoop:96)
    at libunity.ExecutePlayerLoop(NativePlayerLoopSystem*)(ExecutePlayerLoop:176)
    at libunity.PlayerLoop()(PlayerLoop:284)
    at libunity.UnityPlayerLoop()(UnityPlayerLoop:572)
    at libunity.nativeRender(_JNIEnv*, _jobject*)(nativeRender:52)
    at base.oatexec(oatexec)

Sample code

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using System;
using System.IO;
using System.Runtime.InteropServices;
using SoapBoxLabs;

[CreateAssetMenu(fileName = "New Porcupine Manager", menuName = "PorcupineManager")]
public class PorcupineManager : ScriptableObject {
    static string modelPath = "/sdcard/Porcupine/porcupine_params.pv\0";
    string[] keywords = { "/sdcard/Porcupine/Keyword_Files/avocado_android.ppn\0" };
    float[] sensitivity = { .5f };

    public BlankDelegate OnKeywordDetected;

    IntPtr porcupineObject;
    bool porcupineLoaded = false;
    IEnumerator listenCoroutine = null;

    public bool Loaded { get { return porcupineLoaded; } }
    public bool IsRecording { get { return listenCoroutine != null;} }

    [DllImport("pv_porcupine", EntryPoint = "Java_ai_picovoice_porcupine_Porcupine_getFrameLength", CallingConvention = CallingConvention.Cdecl)]
    private static extern int getFrameLength();
    [DllImport("pv_porcupine", EntryPoint = "Java_ai_picovoice_porcupine_Porcupine_getSampleRate", CallingConvention = CallingConvention.Cdecl)]
    private static extern int getSampleRate();
    [DllImport("pv_porcupine", EntryPoint = "Java_ai_picovoice_porcupine_Porcupine_init", CallingConvention = CallingConvention.Cdecl)]
    private static extern IntPtr init(string modelFilePath, string[] keywordFilePaths, float[] sensitivities);
    [DllImport("pv_porcupine", EntryPoint = "Java_ai_picovoice_porcupine_Porcupine_process", CallingConvention = CallingConvention.Cdecl)]
    private static extern int process(IntPtr porcupineAddress, short[] pcm);
    [DllImport("pv_porcupine", EntryPoint = "Java_ai_picovoice_porcupine_Porcupine_delete", CallingConvention = CallingConvention.Cdecl)]
    private static extern void delete(IntPtr porcupineAddress);

    public void Init() {
        Debug.Log(getFrameLength());
        Debug.Log(getSampleRate());

        Debug.Log("Attempting porcupine init: \n" + modelPath + " found? " + File.Exists(modelPath) + "\n"
        + keywords[0] + " found? " + File.Exists(keywords[0]));

        porcupineObject = init(modelPath, keywords, sensitivity);
        porcupineLoaded = true;
    }
}

kenarsa commented 6 years ago

I changed the title of the issue to "Expose JNI method signatures for Android" which I believe is what we need eventually. I don't want to provide the signature of JNI methods in an issue because if we change the API in the future this will break your code with you knowing.

Exposing more APIs on Android is a feature candidate for v1.5. I don't have timing estimates at the moment for this feature or v1.5 release. There are a few things I can think off.

1- What is the standard way of integrating third-party native libraries into Unity on Android? I would think there is a standard way to use Java API? No? I could wrong. This is the easiest solution. My preferred solution. 2- Use an RPi shared object and build on top of that. This is tricky as RPi is linked to a different libc but might work. You can build your own JNI on top of C API. 3- If this is a commercial application contact Picovocie directly and we might be able to allocate some resources to get this done sooner or provide the JNI under NDA ...

rtumelty commented 6 years ago

That's a good reason!

Unity allows you to interact with Java libraries as well as C/C+, so yes, a Java wrapper is an option. Calling a Java wrapper to a JNI wrapper to a C library is very messy though, I'd much prefer to wait until I can interact directly with Porcupine in v1.5, as described in the other issue I linked above, bypassing the JNI.

For the time being, I've done an iOS implementation for the proof of concept, and will be pausing development on Android.

Thanks

kenarsa commented 5 years ago

I've had a chance to look back into this issue. Taking another look at it I think this approach you are trying to take won't work even if we expose the JNI interface. Why? Because each JNI method starts with JNIEnv *env, jobject thiz which is used by Java. You can't possibly provide the correct parameters. The correct way would be to expose actual library API in the .so which is being tracked here https://github.com/Picovoice/Porcupine/issues/73

Closing the issue for now. Reopen if the above doesn't make sense.

Picovoice / porcupine