Android support?

[samolego]

Hi there, would it be possible to use this library in android app?

[samolego]

When adding to android project, it builds fine, but upon loading jllama lib it crashes (loading llama works fine).

I extracted the libraries from app and set the "de.kherud.llama.lib.path" system property.

It also crashes when just manually trying to load library using System#loadLibrary (with Bad JNI version returned from JNI_OnLoad in "/data/app/~~7fH5suWHFaN0djApeTgrxQ==/org.samo_lego.locallm-W9oFa85WAlV_5rcO5ihGJw==/base.apk!/lib/arm64-v8a/libjllama.so": 65537)

Here's the logs, though not so useful as it behaves like it didn't find the library whereas loading failed:

FATAL EXCEPTION: main
                                                                                                    Process: org.samo_lego.locallm, PID: 21224
                                                                                                    java.lang.UnsatisfiedLinkError: No native library found for os.name=Linux-Android, os.arch=aarch64, paths=[/data/app/~~D4qO_eufdtAY_uU8bG46NQ==/org.samo_lego.locallm-NwqKLpgDNDxLFDTLnHZv7w==:/system/lib64:/system_ext/lib64]
                                                                                                        at de.kherud.llama.LlamaLoader.loadNativeLibrary(LlamaLoader.java:141)
                                                                                                        at de.kherud.llama.LlamaLoader.initialize(LlamaLoader.java:66)
                                                                                                        at de.kherud.llama.LlamaModel.<clinit>(LlamaModel.java:29)
                                                                                                        at org.samo_lego.locallm.LMLoader.testText(LMLoader.kt:10)
                                                                                                        at org.samo_lego.locallm.ComposableSingletons$MainActivityKt$lambda-1$1.invoke(MainActivity.kt:39)
                                                                                                        at org.samo_lego.locallm.ComposableSingletons$MainActivityKt$lambda-1$1.invoke(MainActivity.kt:38)
                                                                                                        at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.jvm.kt:108)
                                                                                                        at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.jvm.kt:35)
                                                                                                        at androidx.compose.material3.SurfaceKt$Surface$1.invoke(Surface.kt:132)
                                                                                                        at androidx.compose.material3.SurfaceKt$Surface$1.invoke(Surface.kt:114)
                                                                                                        at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.jvm.kt:108)
                                                                                                        at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.jvm.kt:35)
                                                                                                        at androidx.compose.runtime.CompositionLocalKt.CompositionLocalProvider(CompositionLocal.kt:228)
                                                                                                        at androidx.compose.material3.SurfaceKt.Surface-T9BRK9s(Surface.kt:111)
                                                                                                        at org.samo_lego.locallm.ComposableSingletons$MainActivityKt$lambda-2$1.invoke(MainActivity.kt:35)
                                                                                                        at org.samo_lego.locallm.ComposableSingletons$MainActivityKt$lambda-2$1.invoke(MainActivity.kt:33)
                                                                                                        at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.jvm.kt:108)
                                                                                                        at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.jvm.kt:35)
                                                                                                        at androidx.compose.runtime.CompositionLocalKt.CompositionLocalProvider(CompositionLocal.kt:228)
                                                                                                        at androidx.compose.material3.TextKt.ProvideTextStyle(Text.kt:360)
                                                                                                        at androidx.compose.material3.MaterialThemeKt$MaterialTheme$1.invoke(MaterialTheme.kt:81)
                                                                                                        at androidx.compose.material3.MaterialThemeKt$MaterialTheme$1.invoke(MaterialTheme.kt:80)
                                                                                                        at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.jvm.kt:108)
                                                                                                        at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.jvm.kt:35)
                                                                                                        at androidx.compose.runtime.CompositionLocalKt.CompositionLocalProvider(CompositionLocal.kt:228)
                                                                                                        at androidx.compose.material3.MaterialThemeKt.MaterialTheme(MaterialTheme.kt:73)
                                                                                                        at org.samo_lego.locallm.ui.theme.ThemeKt.LocalLMTheme(Theme.kt:65)
                                                                                                        at org.samo_lego.locallm.ComposableSingletons$MainActivityKt$lambda-3$1.invoke(MainActivity.kt:33)
                                                                                                        at org.samo_lego.locallm.ComposableSingletons$MainActivityKt$lambda-3$1.invoke(MainActivity.kt:32)
                                                                                                        at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.jvm.kt:108)
                                                                                                        at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.jvm.kt:35)
                                                                                                        at androidx.compose.ui.platform.ComposeView.Content(ComposeView.android.kt:428)
                                                                                                        at androidx.compose.ui.platform.AbstractComposeView$ensureCompositionCreated$1.invoke(ComposeView.android.kt:252)
                                                                                                        at androidx.compose.ui.platform.AbstractComposeView$ensureCompositionCreated$1.invoke(ComposeView.android.kt:251)
                                                                                                        at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.jvm.kt:108)
                                                                                                        at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.jvm.kt:35)
                                                                                                        at androidx.compose.runtime.CompositionLocalKt.CompositionLocalProvider(CompositionLocal.kt:228)
                                                                                                        at androidx.compose.ui.platform.CompositionLocalsKt.ProvideCommonCompositionLocals(CompositionLocals.kt:195)
                                                                                                        at androidx.compose.ui.platform.AndroidCompositionLocals_androidKt$ProvideAndroidCompositionLocals$3.invoke(AndroidCompositionLocals.android.kt:119)
                                                                                                        at androidx.compose.ui.platform.AndroidCompositionLocals_androidKt$ProvideAndroidCompositionLocals$3.invoke(AndroidCompositionLocals.android.kt:118)
2024-01-03 17:34:36.671 21224-21224 AndroidRuntime          org.samo_lego.locallm                E      at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.jvm.kt:108)
                                                                                                        at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.jvm.kt:35)
                                                                                                        at androidx.compose.runtime.CompositionLocalKt.CompositionLocalProvider(CompositionLocal.kt:228)
                                                                                                        at androidx.compose.ui.platform.AndroidCompositionLocals_androidKt.ProvideAndroidCompositionLocals(AndroidCompositionLocals.android.kt:110)
                                                                                                        at androidx.compose.ui.platform.WrappedComposition$setContent$1$1$2.invoke(Wrapper.android.kt:158)
                                                                                                        at androidx.compose.ui.platform.WrappedComposition$setContent$1$1$2.invoke(Wrapper.android.kt:157)
                                                                                                        at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.jvm.kt:108)
                                                                                                        at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.jvm.kt:35)
                                                                                                        at androidx.compose.runtime.CompositionLocalKt.CompositionLocalProvider(CompositionLocal.kt:228)
                                                                                                        at androidx.compose.ui.platform.WrappedComposition$setContent$1$1.invoke(Wrapper.android.kt:157)
                                                                                                        at androidx.compose.ui.platform.WrappedComposition$setContent$1$1.invoke(Wrapper.android.kt:142)
                                                                                                        at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.jvm.kt:108)
                                                                                                        at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.jvm.kt:35)
                                                                                                        at androidx.compose.runtime.ActualJvm_jvmKt.invokeComposable(ActualJvm.jvm.kt:78)
                                                                                                        at androidx.compose.runtime.ComposerImpl.doCompose(Composer.kt:3340)
                                                                                                        at androidx.compose.runtime.ComposerImpl.composeContent$runtime_release(Composer.kt:3273)
                                                                                                        at androidx.compose.runtime.CompositionImpl.composeContent(Composition.kt:588)
                                                                                                        at androidx.compose.runtime.Recomposer.composeInitial$runtime_release(Recomposer.kt:1013)
                                                                                                        at androidx.compose.runtime.CompositionImpl.setContent(Composition.kt:520)
                                                                                                        at androidx.compose.ui.platform.WrappedComposition$setContent$1.invoke(Wrapper.android.kt:142)
                                                                                                        at androidx.compose.ui.platform.WrappedComposition$setContent$1.invoke(Wrapper.android.kt:133)
                                                                                                        at androidx.compose.ui.platform.AndroidComposeView.setOnViewTreeOwnersAvailable(AndroidComposeView.android.kt:1191)
                                                                                                        at androidx.compose.ui.platform.WrappedComposition.setContent(Wrapper.android.kt:133)
                                                                                                        at androidx.compose.ui.platform.WrappedComposition.onStateChanged(Wrapper.android.kt:183)
                                                                                                        at androidx.lifecycle.LifecycleRegistry$ObserverWithState.dispatchEvent(LifecycleRegistry.kt:314)
                                                                                                        at androidx.lifecycle.LifecycleRegistry.addObserver(LifecycleRegistry.kt:192)
                                                                                                        at androidx.compose.ui.platform.WrappedComposition$setContent$1.invoke(Wrapper.android.kt:140)
                                                                                                        at androidx.compose.ui.platform.WrappedComposition$setContent$1.invoke(Wrapper.android.kt:133)
                                                                                                        at androidx.compose.ui.platform.AndroidComposeView.onAttachedToWindow(AndroidComposeView.android.kt:1266)
                                                                                                        at android.view.View.dispatchAttachedToWindow(View.java:21980)
                                                                                                        at android.view.ViewGroup.dispatchAttachedToWindow(ViewGroup.java:3490)
                                                                                                        at android.view.ViewGroup.dispatchAttachedToWindow(ViewGroup.java:3497)
                                                                                                        at android.view.ViewGroup.dispatchAttachedToWindow(ViewGroup.java:3497)
                                                                                                        at android.view.ViewGroup.dispatchAttachedToWindow(ViewGroup.java:3497)
                                                                                                        at android.view.ViewGroup.dispatchAttachedToWindow(ViewGroup.java:3497)
                                                                                                        at android.view.ViewRootImpl.performTraversals(ViewRootImpl.java:3014)
                                                                                                        at android.view.ViewRootImpl.doTraversal(ViewRootImpl.java:2465)
                                                                                                        at android.view.ViewRootImpl$TraversalRunnable.run(ViewRootImpl.java:9309)
                                                                                                        at android.view.Choreographer$CallbackRecord.run(Choreographer.java:1339)
                                                                                                        at android.view.Choreographer$CallbackRecord.run(Choreographer.java:1348)
                                                                                                        at android.view.Choreographer.doCallbacks(Choreographer.java:952)
                                                                                                        at android.view.Choreographer.doFrame(Choreographer.java:882)
                                                                                                        at android.view.Choreographer$FrameDisplayEventReceiver.run(Choreographer.java:1322)
                                                                                                        at android.os.Handler.handleCallback(Handler.java:958)
                                                                                                        at android.os.Handler.dispatchMessage(Handler.java:99)
                                                                                                        at android.os.Looper.loopOnce(Looper.java:205)
                                                                                                        at android.os.Looper.loop(Looper.java:294)
2024-01-03 17:34:36.672 21224-21224 AndroidRuntime          org.samo_lego.locallm                E      at android.app.ActivityThread.main(ActivityThread.java:8172)
                                                                                                        at java.lang.reflect.Method.invoke(Native Method)
                                                                                                        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:552)
                                                                                                        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:878)

[samolego]

Ah, seems like Android needs JNI_VERSION_1_6 to be returned ...

[kherud]

Hi @samolego thanks for the issue. I think returning JNI_VERSION_1_6 shouldn't be a problem, I will look at it later today. The pre-built shared libraries don't support android yet, I think, so your best bet would be to compile them yourself (see https://github.com/kherud/java-llama.cpp#setup-required). However, I'll also have a look if I can easily add android shared libraries.

[samolego]

Thanks! Would it be possible to load libraries using System#loadLibrary as well, if detected OS is Android? That allows for loading libs from APK directly, without the need to specify path.

[samolego]

I'm not sure if this is android related, but after editing the source (so I got it to load), this error shows up: https://pastebin.com/WjG8bFAE

java_vm_ext.cc:591] JNI DETECTED ERROR IN APPLICATION: jfieldID int de.kherud.llama.InferenceParameters.nPredict not valid for an object of class de.kherud.llama.ModelParameters
    java_vm_ext.cc:591]     in call to GetIntField
    java_vm_ext.cc:591]     from void de.kherud.llama.LlamaModel.loadModel(java.lang.String, de.kherud.llama.ModelParameters)

(Loading is taken from readme example):

        ModelParameters modelParams = new ModelParameters()
                .setNGpuLayers(43);
        InferenceParameters inferParams = new InferenceParameters()
                .setTemperature(0.7f)
                .setPenalizeNl(true)
                .setMirostat(InferenceParameters.MiroStat.V2)
                .setAntiPrompt("\n");

[samolego]

May I ask why escape field is getting assigned the nPredict value?

https://github.com/kherud/java-llama.cpp/blob/643dc4f079ca07d8d096410e486bf4b3d5abfc1d/src/main/cpp/jllama.cpp#L953 Java-llama: https://github.com/kherud/java-llama.cpp/blob/643dc4f079ca07d8d096410e486bf4b3d5abfc1d/src/main/java/de/kherud/llama/InferenceParameters.java#L20-L21

From llama.cpp:

    bool escape            = false; // escape "\n", "\r", "\t", "\'", "\"", and "\\"

Aren't those different? Sorry for comment spam :sweat_smile:

[kherud]

May I ask why escape field is getting assigned the nPredict value?

Probably a copy paste fail 😬 I'll look into it and fix it too, thanks! I think escape is a parameter mainly used by the original llama.cpp library for CLI input, so I will probably remove it.

Would it be possible to load libraries using System#loadLibrary as well, if detected OS is Android? That allows for loading libs from APK directly, without the need to specify path.

Yes, the intended way for the libraries to be automatically found is to put them in a platform specific resources directory. The correct directory is determined by the OSInfo class, I think here it should be src/main/resources/de/kherud/llama/Linux-Android/arm.

Also, this class is responsible for loading the shared libraries. It should definitely be possible to extend its behavior to search in the APK for android. There already is OSInfo#isAndroid() to check if the current platform is Android.

[samolego]

Yeah, for my dev purposes I modified LlamaLoader#loadNativeLibrary so that it has:

        if (OSInfo.isAndroid()) {
            try {
                System.loadLibrary(name);  // loadLibrary can load directly from packed apk file automatically, if c++ sources are added
                return;
            } catch (UnsatisfiedLinkError e) {  // handle
                        }
                }

(as I included your repo as a submodule). This way the library loads from where it's packed to by gradle. This is useful as it only compiles for the architecture you're building the apk for. I'm still not getting lucky though as some native errors started to pop up. If you have time (and motivation), I'd greatly appreciate if you'd try to get it working on android :)

[kherud]

I just released version 2.3.2, which has a pre-built shared library for android arm64 (see attachment also) and removed the mentioned parameter. I don't think the error came from JNI_VERSION_1_1 and I'm hesitant to increase it to JNI_VERSION_1_6 for compatibility. Do you have any sources why this is necessary?

Unfortunately, I can't really test if every runs correctly on android. So feel free to report back if you still have problems. Your best bet is still to compile the library for android yourself, I think, since the pre-built one isn't performance optimized for your CPU.

Linux-Android.zip

[samolego]

I understand the compatibility hesitance, though I think it came from it, as it started working when changing 1_1 -> 1_6

I'll test this and report back though :). Thank you!

[samolego]

2.3.2 still fails on library loading: https://pastebin.com/BpyXTGVe

What's strange to me is that when I open the built apk file, in de/kherud/llama there's only Mac/ and Windows/ dirs ..?

[samolego]

The 1_6 version might be device specific, I'm not sure. The log says (with 1_1): java.lang.UnsatisfiedLinkError: Bad JNI version returned from JNI_OnLoad in "/data/app/~~hUolbigPb4piWEUB_KFU3A==/org.samo_lego.locallm-JIQvJRf4_WXRK75tvKloGg==/base.apk!/lib/arm64-v8a/libjllama.so": 65537 but if I change it to 1_6 it works ... I have Android 14

Edit: found this: https://groups.google.com/g/android-ndk/c/ukQBmKJH2eM Indeed, setting the return value to version 1.2 and 1.4 works as well :P

https://cs.android.com/android/platform/superproject/main/+/main:art/runtime/jni/java_vm_ext.cc;drc=24de61c5bf4cf700aef8739266ebae2db7fdb0ec;l=70 official source

[samolego]

Anyway, after loading the library on my own, I'm getting some exceptions with JNI from jllama_log_callback: env->CallVoidMethod(g_log_callback, m_biconsumer_accept, java_log_level, java_text); This causes crash: https://pastebin.com/Xv4b2Ngx

jobject is an invalid local reference: 0x69d5eed18d (popped reference at index 99 in a table of size 12)
    java_vm_ext.cc:591]     in call to CallVoidMethodV

[samolego]

The upper error was caused by log levels not being valid references. I changed the assignments:

    o_log_level_debug = env->GetStaticObjectField(c_log_level, f_log_level_debug);
    o_log_level_info = env->GetStaticObjectField(c_log_level, f_log_level_info);
    o_log_level_warn = env->GetStaticObjectField(c_log_level, f_log_level_warn);
    o_log_level_error = env->GetStaticObjectField(c_log_level, f_log_level_error);

to

    o_log_level_debug = (jobject)env->NewGlobalRef(env->GetStaticObjectField(c_log_level, f_log_level_debug));
    o_log_level_info = (jobject)env->NewGlobalRef(env->GetStaticObjectField(c_log_level, f_log_level_info));
    o_log_level_warn = (jobject)env->NewGlobalRef(env->GetStaticObjectField(c_log_level, f_log_level_warn));
    o_log_level_error = (jobject)env->NewGlobalRef(env->GetStaticObjectField(c_log_level, f_log_level_error));

and added their deletion in JNI_OnUnload

As this is my first time doing JNI; is this allright? Do you perhaps know why it would crash?

[samolego]

All of this (together with #38) makes it load successfully in my app 🎉 ! Now I'm onto figuring out why it seems stuck on generation :P

[samolego]

The upper error was caused by log levels not being valid references. I changed the assignments:

    o_log_level_debug = env->GetStaticObjectField(c_log_level, f_log_level_debug);
    o_log_level_info = env->GetStaticObjectField(c_log_level, f_log_level_info);
    o_log_level_warn = env->GetStaticObjectField(c_log_level, f_log_level_warn);
    o_log_level_error = env->GetStaticObjectField(c_log_level, f_log_level_error);

to

    o_log_level_debug = (jobject)env->NewGlobalRef(env->GetStaticObjectField(c_log_level, f_log_level_debug));
    o_log_level_info = (jobject)env->NewGlobalRef(env->GetStaticObjectField(c_log_level, f_log_level_info));
    o_log_level_warn = (jobject)env->NewGlobalRef(env->GetStaticObjectField(c_log_level, f_log_level_warn));
    o_log_level_error = (jobject)env->NewGlobalRef(env->GetStaticObjectField(c_log_level, f_log_level_error));

and added their deletion in JNI_OnUnload

As this is my first time doing JNI; is this allright? Do you perhaps know why it would crash?

@kherud do you perhaps know why above thing would be crashing?

[kherud]

Hey @samolego nice observation 👍 I think my initial assumption when writing that code was, that enumeration option objects statically exist once in the JVM, and thus don't need a global reference. The way JNI works, the latter part this is wrong, however. Local references are automatically cleaned up to prevent memory leaks in native code. Thus they aren't persistent across multiple method calls and can lead to segmentation faults if used like in my previous implementation. So even though the object stays the same, local references aren't persistent. That's why creating global references like you did works.

[samolego]

Some more questions regarding android support:

• would you be willing to increase JNI_VERSION_1_1 to JNI_VERSION_1_2? (source: https://cs.android.com/android/platform/superproject/main/+/main:art/runtime/jni/java_vm_ext.cc;drc=24de61c5bf4cf700aef8739266ebae2db7fdb0ec;l=70). If not, is there some way to change return value based on which system it's building for (this might be even better option, but I'm not sure how would one do it)?
• (after above) would it be ok if I make another PR that allows loading the library using System.loadLibrary if detected os is android? This enables loading library from apk file directly. This way, you can add java-llama.cpp as a submodule in android project, link its cmake to gradle and it will build and include the apk file.

[kherud]

would you be willing to increase JNI_VERSION_1_1 to JNI_VERSION_1_2? (source: https://cs.android.com/android/platform/superproject/main/+/main:art/runtime/jni/java_vm_ext.cc;drc=24de61c5bf4cf700aef8739266ebae2db7fdb0ec;l=70). If not, is there some way to change return value based on which system it's building for (this might be even better option, but I'm not sure how would one do it)?

Apparently all the JNI version were released in the respective 1.x JDK. So I think my argument about backwards compatibility is stupid, since supporting Java 1.1 is infeasible anyway. Please go ahead and increase the version.

(after above) would it be ok if I make another PR that allows loading the library using System.loadLibrary if detected os is android? This enables loading library from apk file directly. This way, you can add java-llama.cpp as a submodule in android project, link its cmake to gradle and it will build and include the apk file.

In general, I think the whole library loading code would really benefit from refactoring in its current state. Adding an exception for android wouldn't make it much worse. So please, go ahead. It would be nice if you still keep the current code as a fallback, i.e., first try to load from APK and if that didn't work, use the old procedure.