search

airockchip / rknn-llm

Other
417 stars 36 forks source link

What model to use supporting embeddings? #108

Open vincenzodentamaro opened 4 weeks ago

vincenzodentamaro commented 4 weeks ago

Hi, given that now RKNN-LLM supports embedding, I tried with different models , but no one works with embeddings. Can you point me out a model that supports embedding and is compatible with RKLLM?

Thank you @waydong

waydong commented 2 weeks ago

Hi,

  1. embedding input is primarily used for multimodal models(such as MiniCPMV, https://huggingface.co/openbmb/MiniCPM-V-2_6/tree/main).
  2. For usage instructions, you can refer to this demo(https://github.com/airockchip/rknn-llm/blob/main/rkllm-runtime/examples/rkllm_api_demo/src/multimodel_demo.cpp)
  3. Version 1.1.2 now supports exporting the llm model from MiniCPMV. You can give it a try.