search

DakeQQ / Native-LLM-for-Android

Demonstration of running a native LLM on Android device.
https://dakeqq.github.io/overview/
Apache License 2.0
52 stars 6 forks source link

readme

Native-LLM-for-Android

  1. Demonstration of running a native LLM on Android device. Now support:
    • Qwen2-Instruct: 0.5B, 1.5B ...
    • MiniCPM-DPO/SFT: 1B, 2.7B
    • Yuan2.0: update soon...
    • Gemma2-it: 2B...
    • StableLM2-Chat/Zephyr: update soon...
    • Phi3.5-mini-instruct: 3.8B
  2. The demo models were uploaded to the drive: https://drive.google.com/drive/folders/1E43ApPcOq3I2xvb9b7aOxazTcR3hn5zK?usp=drive_link
  3. After downloading, place the model into the assets folder.
  4. Remember to decompress the *.so zip file stored in the libs/arm64-v8a folder.
  5. The demo models were converted from HuggingFace or ModelScope and underwent code optimizations to achieve extreme execution speed.
  6. Therefore, the inputs & outputs of the demo models are slightly different from the original one.
  7. To better adapt to ONNX Runtime on Android, the export did not use dynamic axes. Therefore, the exported ONNX model may not be optimal for x86_64 systems.
  8. The tokenizer.cpp and tokenizer.hpp files originated from the mnn-llm repository.
  9. To export the model on your own, please go to the 'Export_ONNX' folder, follow the comments to set the folder path, and then execute the ***_Export.py Python script. Next, quantize / optimize the onnx model by yourself.
  10. If use onnxruntime.tools.convert_onnx_models_to_ort to convert to the *.ort format, it will automatically add Cast operators that changes fp16 multiplication to fp32.
  11. The quantization method for the model can be seen in the folder "Do_Quantize".
  12. The q4(uint4) quantization method is not currently recommended because the "MatMulNBits" operator in ONNX Runtime is performing poorly.
  13. See more projects: https://dakeqq.github.io/overview/

    安卓本地运行LLM

  14. 在Android设备上运行本地LLM的演示。目前支持:
    • 通义千问2-Instruct: 0.5B, 1.5B ...
    • MiniCPM-DPO/SFT: 1B, 2.7B
    • 源2.0: update soon...
    • Gemma2-it: 2B
    • StableLM2-Chat/Zephyr: update soon...
    • Phi3.5-mini-instruct: 3.8B
  15. 演示模型已上传至云端硬盘：https://drive.google.com/drive/folders/1E43ApPcOq3I2xvb9b7aOxazTcR3hn5zK?usp=drive_link
  16. 百度: https://pan.baidu.com/s/1NHbUyjZ_VC-o62G13KCrSA?pwd=dake 提取码: dake
  17. 下载后，请将模型文件放入assets文件夹。
  18. 记得解压存放在libs/arm64-v8a文件夹中的*.so压缩文件。
  19. 演示模型是从HuggingFace或ModelScope转换来的，并经过代码优化，以实现极致执行速度。
  20. 因此，演示模型的输入输出与原始模型略有不同。
  21. 为了更好的适配ONNXRuntime-Android，导出时未使用dynamic-axes. 因此导出的ONNX模型对x86_64而言不一定是最优解.
  22. tokenizer.cpp和tokenizer.hpp文件源自mnn-llm仓库。
  23. 想自行导出模型请前往“Export_ONNX”文件夹，按照注释操作设定文件夹路径，然后执行 ***_Export.py的python脚本。下一步，自己动手量化或优化导出的ONNX模型。
  24. 若使用onnxruntime.tools.convert_onnx_models_to_ort转成*.ort格式，它会自动添加Cast算子将fp16乘法转成fp32。
  25. 模型的量化方法可以在文件夹 "Do_Quantize" 中查看。
  26. 现在不建议使用q4(uint4)量化方法, 因为ONNX Runtime的运算符"MatMulNBits"表现不佳。
  27. 看更多項目: https://dakeqq.github.io/overview/

    通义千问 Qwen - 性能 Performance

    OS Device Backend Model Inference
    ( 1024 Context )
    Android 13 Nubia Z50 8_Gen2-CPU
    (X3+A715)    		 Qwen2-1.5B-Instruct
    q8f32    		 20 token/s
    Harmony 4 P40 Kirin_990_5G-CPU
    (2*A76)    		 Qwen2-1.5B-Instruct
    q8f32    		 13 token/s
    Harmony 3 荣耀20S Kirin_810-CPU
    (2*A76)    		 Qwen2-1.5B-Instruct
    q8f32    		 7 token/s

    MiniCPM - 性能 Performance

    OS Device Backend Model Inference
    ( 1024 Context )
    Android 13 Nubia Z50 8_Gen2-CPU
    (X3+A715)    		 MiniCPM-2.7B
    q8f32    		 9.5 token/s
    Harmony 4 P40 Kirin_990_5G-CPU
    (2*A76)    		 MiniCPM-2.7B
    q8f32    		 6 token/s
    Android 13 Nubia Z50 8_Gen2-CPU
    (X3+A715)    		 MiniCPM-1.3B
    q8f32    		 16.5 token/s
    Harmony 4 P40 Kirin_990_5G-CPU
    (2*A76)    		 MiniCPM-1.3B
    q8f32    		 11 token/s

    源 Yuan - 性能 Performance

    OS Device Backend Model Inference
    ( 1024 Context )
    Android 13 Nubia Z50 8_Gen2-CPU
    (X3+A715)    		 Yuan2.0-2B-Mars-hf
    q8f32    		 12 token/s
    Harmony 4 P40 Kirin_990_5G-CPU
    (2*A76)    		 Yuan2.0-2B-Mars-hf
    q8f32    		 6.5 token/s

    Gemma - 性能 Performance

    OS Device Backend Model Inference
    ( 1024 Context )
    Android 13 Nubia Z50 8_Gen2-CPU
    (X3+A715)    		 Gemma1.1-it-2B
    q8f32    		 16 token/s

    StableLM - 性能 Performance

    OS Device Backend Model Inference
    ( 1024 Context )
    Android 13 Nubia Z50 8_Gen2-CPU
    (X3+A715)    		 StableLM2-1.6B-Chat
    q8f32    		 17.8 token/s
    Harmony 4 P40 Kirin_990_5G-CPU
    (2*A76)    		 StableLM2-1.6B-Chat
    q8f32    		 11 token/s
    Harmony 3 荣耀20S Kirin_810-CPU
    (2*A76)    		 StableLM2-1.6B-Chat
    q8f32    		 5.5 token/s

    Phi - 性能 Performance

    OS Device Backend Model Inference
    ( 1024 Context )
    Android 13 Nubia Z50 8_Gen2-CPU
    (X3+A715)    		 Phi2-2B-Orange-V2
    q8f32    		 9.5 token/s
    Harmony 4 P40 Kirin_990_5G-CPU
    (2*A76)    		 Phi2-2B-Orange-V2
    q8f32    		 5.8 token/s

    演示结果 Demo Results

    (Qwen2-1.5B / 1024 Context)
    Demo Animation