search

Meituan-AutoML / MobileVLM

Strong and Open Vision Language Assistant for Mobile Devices
Apache License 2.0
958 stars 65 forks source link

inference on iPhone #2

Closed tstanek390 closed 6 months ago

tstanek390 commented 8 months ago

Hello, thanks for sharing these very promising LLMs. I'd like to run it on iOS, but as I'm a noob at coding and stuff, I don't really know how to start. I'm an healtcare professional and would love to give it a try. Any hints/tips ? Thanks a lot and all the best to the new year! Teodor

huyiming2018 commented 8 months ago

Maybe you can refer to this example from lamma.cpp community: https://github.com/ggerganov/llama.cpp/discussions/4508

tstanek390 commented 8 months ago

Thx, but I guess I would need GGUF version of the model to run it with LLama.cpp..?

YangYang-DLUT commented 8 months ago

You can convert the model to GGUF follow the instruction of https://github.com/ggerganov/llama.cpp/tree/master/examples/llava. The detailed deployment instruction is in process and will be released soon.

YangYang-DLUT commented 7 months ago

We have updated README.md with the deployment instruction on Android devices with customized llama.cpp. Have a try!

The inference plan on iPhone is on schedule but not proposed yet. Any ideas?

jasonw247 commented 7 months ago

Thanks for adding this! Are you planning on submitting a PR to llama.cpp, or keeping the forked repo?

YangYang-DLUT commented 7 months ago

We have submitted a PR to llama.cpp. We want to get the support of open-source communities, since that there are a lot space for optimization of the deployment on mobile devices.

er-muyue commented 6 months ago

Hi, we are closing this issue due to the inactivity. Hope your question has been resolved. If you have any further concerns, please feel free to re-open it or open a new issue. Thanks!