iOS LLM Task SDK is NOT initialized successfully via KMP

[2BAB]

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

Yes

OS Platform and Distribution

iOS 15

MediaPipe Tasks SDK version

0.10.14

Task name (e.g. Image classification, Gesture recognition etc.)

LLM Inference

Programming Language and version (e.g. C++, Python, Java)

Kotlin, Objective-C, Swift

Describe the actual behavior

It crashed (NPE) right after "Metal LLM tokens initialized" got printed.

Describe the expected behaviour

Should be initialized normally. Can add a new init method with error catching inside and returning nullable error string.

Standalone code/steps you may have used to try to get what you need

1. Integrate iOS SDK into a Kotlin Multiplatform project using Cocoapods Gradle Plugin:

kotlin {    

    listOf(
        iosX64(),
        iosArm64(),
        iosSimulatorArm64()
    ).forEach {
        it.binaries.framework {
            baseName = "Mediapiper"
            isStatic = true
        }
        it.binaries.all {
            linkerOpts("-L/usr/lib/swift")
            linkerOpts("-rpath", "/usr/lib/swift")
            val aicPathSuffix = when (this.target.konanTarget) {
                KonanTarget.IOS_ARM64 -> "ios-arm64"
                KonanTarget.IOS_X64, KonanTarget.IOS_SIMULATOR_ARM64 -> "ios-arm64_x86_64-simulator"
                else -> null
            }
            aicPathSuffix?.let { p ->
                listOf(
                    "MediaPipeTasksGenAIC",
                    "MediaPipeTasksGenAI"
                ).forEach { f ->
                    linkerOpts("-framework", f, "-F../iosApp/Pods/$f/frameworks/$f.xcframework/$p")
                }
                val swiftPathSuffix = when (this.target.konanTarget) {
                    KonanTarget.IOS_ARM64 -> "iphoneos"
                    KonanTarget.IOS_X64, KonanTarget.IOS_SIMULATOR_ARM64 -> "iphonesimulator"
                    else -> null
                }
                swiftPathSuffix?.let { sp ->
                    val swiftPathPrefix =
                        "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib"
                    linkerOpts("-L$swiftPathPrefix/swift/$sp")
                    linkerOpts("-rpath", "$swiftPathPrefix/swift-5.0/$sp")
                }
            }
        }
    }

    cocoapods {
        name = "Mediapiper"

        version = "1.0.1"
        ios.deploymentTarget = "15"

        summary = "Mediapiper"
        homepage = "https://github.com/2BAB/Mediapiper"

        pod("MediaPipeTasksGenAIC") {
            version = "0.10.14"
            extraOpts += listOf("-compiler-option", "-fmodules")
        }
        pod("MediaPipeTasksGenAI") {
            version = "0.10.14"
            extraOpts += listOf("-compiler-option", "-fmodules")
        }

    }

}

2. Import the MPPLLMInference that generated by Kotlin Interop from step 1:

import cocoapods.MediaPipeTasksGenAI.MPPLLMInference
import cocoapods.MediaPipeTasksGenAI.MPPLLMInferenceOptions

3. Try to initialize it (the same way as what official sample did）:

    init {
        val modelPath = NSBundle.mainBundle.pathForResource("gemma-2b-it-gpu-int4", "bin")

        val options = MPPLLMInferenceOptions(modelPath!!)
        options.setModelPath(modelPath!!)
        options.setMaxTokens(1024)
        options.setTopk(40)
        options.setTemperature(0.8f)
        options.setRandomSeed(102)
        // By default it's nullable
        inference = MPPLLMInference(options, null) // NPE(NullPointerException) throws here! Right after a SDK internal log "Metal LLM tokens initialized" printed.
        // Or in another way, trying to pass the error handler
        memScoped {
          var error = alloc<ObjCObjectVar<NSError?>>()
          val inference = MPPLLMInference(options, error.ptr) // The same NPE was thrown.
        }
    }

This may come from the limitation of KMP interop (https://kotlinlang.org/docs/native-objc-interop.html#errors-and-exceptions), since the constructor signature is @objc public init(options: Options) throws {...}.

https://github.com/google-ai-edge/mediapipe/blob/cdc08d04ee83f36085894e3a24cb61c949f56b14/mediapipe/tasks/ios/genai/inference/sources/LlmInference.swift#L56

A full example for this issue can be found from below:

https://github.com/2BAB/MediaPiper/blob/ios-sdk-init-crash/app/src/iosMain/kotlin/me.xx2bab.mediapiper/llm/LLMOperator.ios.kt#L50

Other info / Complete Logs

No response

[kuaashish]

Hi @2BAB,

Could you please provide the complete standalone code or the specific steps you are following from our documentation to help us better understand the issue? If needed, we will reproduce it on our end.

Thank you!!

[2BAB]

@kuaashish updated the standalone code snippet and bolded the full example link.