cpu model not supported?

hemanth commented 5 months ago

llm_inference on local is throwing the below error for gemma-2b-it-cpu-int8.bin only gpu backend type is supported?

calculator_graph.cc:892] INVALID_ARGUMENT: CalculatorGraph::Run() failed: 
Calculator::Open() for node "LlmGpuCalculator" failed: Please use a tensorflow lite model with gpu backend type, while the current model's backend type is: cpu
=== Source Location Trace: === 
third_party/odml/infra/genai/inference/calculators/llm_gpu_calculator.cc:279
third_party/mediapipe/framework/calculator_node.cc:560
_emscripten_errn @ genai_wasm_internal.js:9
$func8282 @ genai_wasm_internal.wasm:0x86abdc
$func4251 @ genai_wasm_internal.wasm:0x509788
$func2510 @ genai_wasm_internal.wasm:0x2780ff
$func200 @ genai_wasm_internal.wasm:0x95eb
$func2446 @ genai_wasm_internal.wasm:0x26c227
$qd @ genai_wasm_internal.wasm:0x596bad
ret.<computed> @ genai_wasm_internal.js:9
Module._waitUntilIdle @ genai_wasm_internal.js:9
finishProcessing @ tasks-genai:7
finishProcessing @ tasks-genai:7
Wr @ tasks-genai:7
(anonymous) @ tasks-genai:7

akashthakur4553 commented 2 months ago

Got the answer? i'm facing the same error with the int4-cpu model

Linchenn commented 2 weeks ago

Are you guys running this on Web? or Android? Would like to know your platforms?

akashthakur4553 commented 2 weeks ago

web

akashthakur4553 commented 2 weeks ago

I think it happens when i completely drain my gpu , so sometimes it runs sometimes it doesn't

Linchenn commented 1 week ago

The LLM Inference Task only supports GPU models on Web, gemma-2b-it-cpu-***.bin models are not supported.

"For web applications, the model must be GPU-compatible." https://ai.google.dev/edge/mediapipe/solutions/genai/llm_inference/web_js#model

google-ai-edge / mediapipe-samples

cpu model not supported? #344