phi 3.5 moe usage - Githubissues

foldl / chatllm.cpp

Pure C++ implementation of several models for real-time chatting on your computer (CPU)

MIT License

375 stars 29 forks source link

phi 3.5 moe usage #31

Open GlasslessPizza opened 2 months ago

GlasslessPizza commented 2 months ago

The model_downloader.py script doesn't list the recently supported phi 3.5 moe. I'd like to also know if it's ok to use the v0.3 release from Jul 6 as-is to run it. Thanks!

ayttop commented 2 months ago

microsoft/Phi-3.5-MoE-instruct/??????????????????????????

ayttop commented 2 months ago

how to convert microsoft/Phi-3.5-MoE-instruct to gguf?

foldl commented 2 months ago

@GlasslessPizza You can try the latest v0.4 or build it yourself.

You can use -m :phi3.5:moe in Python binding.

foldl commented 2 months ago

@ayttop Sorry, GGUF is not supported. This project uses something like the old GGML file format.

ayttop commented 2 months ago

Can the form be converted to ggml and any body convert it to gguf>?