search

google-ai-edge / mediapipe

Cross-platform, customizable ML solutions for live and streaming media.
https://ai.google.dev/edge/mediapipe
Apache License 2.0
27.45k stars 5.15k forks source link

Add GPU support to LLM C API #5305

Closed huazhusun closed 5 days ago

huazhusun commented 6 months ago

Here I use bazel build --config=android_arm64 //mediapipe/tasks/java/com/google/mediapipe/tasks/genai:libllm_inference_engine_jni.so to get libllm_inference_engine_jni.so , I use it in my application, it's crashed. 2024-04-09 18:25:38.304 3468-3678 native com.example.scigemma A F0000 00:00:1712658338.304137 3678 llm_inference_engine_cpu.cc:263] Failed to create session: INTERNAL: ; RET_CHECK failure (mediapipe/tasks/cc/genai/inference/c/llm_inference_engine_cpu.cc:180) (backend)==("cpu")

However, I using https://mvnrepository.com/artifact/com.google.mediapipe/tasks-genai , it's success.

It's same error when using the file tasks_genai.aar built with bazel build -c opt --config=android_arm64 --linkopt="-s" //mediapipe/tasks/java/com/google/mediapipe/tasks/genai:tasks_genai

by the way, the file of " tasks_genai.aar" size I built is 9,823,664, the file size in maven is 5,553,876

Are there some difference ? My compiling steps are: git clone https://github.com/google/mediapipe.git -b v0.10.11 mediapipe-v0.10.11 git checkout e135531 sh ./setup_opencv.sh bash ./setup_android_sdk_and_ndk.sh ~/Android/Sdk ~/Android/Ndk r21 [--accept-licenses] bazel build -c opt --config=android_arm64 --linkopt="-s" //mediapipe/tasks/java/com/google/mediapipe/tasks/genai:tasks_genai

Version: mediapipe-0.10.11 TAG: e135531 bazel version 6.1.1 Android Version: 10

huazhusun commented 6 months ago

In funtion "LlmInferenceEngine_CreateSession_Helper", It always check the backend as "CPU", why?

MP_ASSIGN_OR_RETURN(auto backend, model_data->ReadMetadata( mediapipe::tasks::genai::llm_utils::kLlmBackendName)); RET_CHECK_EQ(backend, "cpu"); Does it mean that LlmInferenceEngine only support CPU?

kuaashish commented 6 months ago

Hi @huazhusun,

Could you confirm whether you have successfully installed the MediaPipe framework from the provided source here? have you been able to run the Hello World example as instructed in the same source?

Thank you!!

schmidt-sebastian commented 6 months ago

Hi! Sorry this a new release and we still trying to tie up loose ends. For now, our open source code only allows us to run CPU models, while the Maven package include GPU support. It looks like you are running a GPU model on the CPU path. We hope to have a better story soon.

schmidt-sebastian commented 5 days ago

Closing this for now, but please follow our blog announcements when we announce public support. Thank you!