search

mlc-ai / mlc-llm

Universal LLM Deployment Engine with ML Compilation
https://llm.mlc.ai/
Apache License 2.0
18.67k stars 1.52k forks source link

Phi 3 128 #2205

Closed sebastienbo closed 3 months ago

sebastienbo commented 4 months ago

🚀 Feature

Introduce phi 3 mini 128k instruct

https://huggingface.co/microsoft/Phi-3-mini-128k-instruct

Mini can run on phones (there is a 4k version and 128k version)

Motivation

Alternatives

Additional context

moondiy commented 4 months ago

r u run phi3 ok?

sebastienbo commented 4 months ago

r u run phi3 ok?

What do you mean?

sebastienbo commented 4 months ago

The 128k version could allow us to analyze a pdf from the phone. the 4k version would probably be less cpu intensive and would work out of the box.

moondiy commented 4 months ago

sorry.i want to known that are yiu run phi3 mini model on the phone?i i run it with onnxruntime genai but it's too slow

sebastienbo commented 4 months ago

Well I use phi2 all the time using ml. Android on my phone with Snapdragon 2nd gen and it works really good and smooth. That is why I would like to know how I can put phi 3 on mlc.

kripper commented 4 months ago

Have you tried following the instructions for building new models?

indiejoseph commented 4 months ago

Tried following the instructions, and got the error:

ValueError: Unknown model type: phi3. Available ones: ['llama', 'mistral', 'gemma', 'gpt2', 'mixtral', 'gpt_neox', 'gpt_bigcode', 'phi-msft', 'phi', 'qwen', 'qwen2', 'stablelm', 'baichuan', 'internlm', 'rwkv5', 'orion', 'llava', 'rwkv6', 'chatglm', 'eagle']
kripper commented 4 months ago

I would try to add the string "phi3" to the list

indiejoseph commented 4 months ago

@kripper thanks for your advice, the MODELS is not really just a list with a model name, it is a dict with the model name as key and the corresponding loader and implementation.

I've tried specify the --model-type phi-msft but without luck.

kripper commented 4 months ago

I would first research if the model loader is compatible with previous versions. If so, map the ID to the existing loader. Otherwise, create a new loader and have fun hacking, ie:

Always prefere to share a same code base for the old and new model and use conditions (if(version == 1) {...} else if(version == 3) {...}).

unmotivatedgene commented 4 months ago

Once this is figured out please update the android app apk with phi3. ❤️

MasterJH5574 commented 3 months ago

Folks, we have supported Phi-3 mini 4k/128k and you can find the pre-converted models at https://huggingface.co/mlc-ai. Phi-3 is available in the Android app and iOS app now.

Closing this issue due to the completeness.

sebastienbo commented 3 months ago

I tried it, but it was impossible to ask any questions, the android app is crashing so badly that even my whole phone became unstable and crashed.

It's totally unusable on a samsung flip 5

Has anyone else tried it?

On top of that, the phi 2 is gone, so I have nothing left now.

On Tue, 11 Jun 2024 at 21:56, Ruihang Lai @.***> wrote:

Folks, we have supported Phi-3 mini 4k/128k and you can find the pre-converted models at https://huggingface.co/mlc-ai. Phi-3 is available in the Android app and iOS app now.

Closing this issue due to the completeness.

— Reply to this email directly, view it on GitHub https://github.com/mlc-ai/mlc-llm/issues/2205#issuecomment-2161504939, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB7MPQUCHPEOVIMGPUO5MNLZG5I45AVCNFSM6AAAAABGVI2W4SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRRGUYDIOJTHE . You are receiving this because you authored the thread.Message ID: @.***>