search

huggingface / candle

Minimalist ML framework for Rust
Apache License 2.0
13.79k stars 751 forks source link

Request: Please add support in examples for the new MS/Phi-3 family #2119

Closed a-agmon closed 3 weeks ago

a-agmon commented 3 weeks ago

Hi, Microsoft just released the new set of Phi 3 models (inc quantized models) that are supposed to be a game changer in performance and quality in memory/cpu limited environments. Im struggling a bit to make it work with Candle and I have a feeling its related to something in the model config that also changed. I think supporting it with example can be tremendously valuable to the HF community.

https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-gguf

Thanks

LaurentMazare commented 3 weeks ago

Support for the quantized version is in #2118 and already works pretty well, we'll also add support for the non quantized version hopefully very soon.

a-agmon commented 3 weeks ago

Fantastic. Thanks @LaurentMazare