search

cdiddy77 / react-native-llm-mediapipe

Run an LLM on iOS & Android devices using React Native
MIT License
73 stars 5 forks source link

GPU Requirements and example app #16

Open EssamWisam opened 1 month ago

EssamWisam commented 1 month ago

I was quite impressed to find that this exists. Thank you a lot for this effort.

I was wondering if you @cdiddy77 have an idea about whether a particular model, after download, is likely to run smoothly over most devices (while giving okayish responses). I have tried two models via https://llm.mlc.ai/docs/deploy/android.html and they make my S22 Ultra heat up severely and potentially crash.

Also, it would be great if there is a demo apk here similar to the one in mlc docs.

Thank you in advance.

cdiddy77 commented 1 month ago

I can really only speak for gemma2b-int4 . It runs pretty well on my samsung galaxy s23 . On my iphone 13 pro it crashes frequently. The phone gets warm but I wouldn't say its terrible.