Performance Issue On Android

BrutalCoding / aub.ai

AubAI brings you on-device gen-AI capabilities, including offline text generation and more, directly within your app.

GNU Affero General Public License v3.0

150 stars 12 forks source link

Hello,

First I'll say, really impressed by this library and looking forward to TTS!

I ran the example project on my android pixel 7 (Same one you used) and I am not seeing the same performance that was presented in the video here https://www.youtube.com/watch?v=SBaSpwXRz94 . I am getting about 1 word every 20 seconds.

I used the tinyllama-1.1b-chat-v1.0.Q2_K.gguf model found here. https://huggingface.co/TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF/tree/main

I tried on a few more models and got the same issue.

Was there a specific model needed to achieved this? Or any specific configuration?

Run command: flutter run -d 28301FDH200MY4 --release

Device Info:

[
  {
    "name": "Pixel 7",
    "id": "28301FDH200MY4",
    "isSupported": true,
    "targetPlatform": "android-arm64",
    "emulator": false,
    "sdk": "Android 14 (API 34)",
    "capabilities": {
      "hotReload": true,
      "hotRestart": true,
      "screenshot": true,
      "fastStart": true,
      "flutterExit": true,
      "hardwareRendering": false,
      "startPaused": true
    }
  }
]

Hi there,

No, I did not use a specific model to achieve this. The one you suggested should work just as fast, but I remember that one of my commits (where I updated llama.cpp) slowed down the speed a lot to an unusable state. Later one, I did update binaries for llama.cpp again and that I believe fixed it.

Are you using this git branch directly (bad instructions, but more up-to-date) or the pub.dev package (easy to install, but outdated)?

Side note: Not saying you're wrong, as I haven't verified it on my Pixel 7 now for a good month give or take. I have shifted focus from working on this project to fixing my financial situation since Dec 2023/Jan 2024. I shared my personal life update here last night: https://github.com/BrutalCoding/aub.ai/discussions/16. Long story short: It's a bit hectic but I got a new job now, and I'd like to continue contributing to this project and keep things up-to-date. Just need to make sure that my open-source contributions and my contributions towards the commercial party is clearly communicated about and agreed upon beforehand.

BrutalCoding / aub.ai

Performance Issue On Android #12