search

huggingface / candle

Minimalist ML framework for Rust
Apache License 2.0
13.79k stars 751 forks source link

Support for Microsoft Phi-3 128k context length #2123

Open niutech opened 2 weeks ago

niutech commented 2 weeks ago

Please allow to use microsoft/Phi-3-mini-128k-instruct model in the candle-phi example, which uses the LongRope scaling. Thanks!

EricLBuehler commented 2 weeks ago

@niutech, we support 128k context length on [mistral.rs]() which you can run with cargo run --release --features ... -- -i plain -m microsoft/Phi-3-mini-128k-instruct -a phi3.

For reference, here is our implementation: https://github.com/EricLBuehler/mistral.rs/blob/6334b30fdf6447fa787dcbedb032fb825c22ae1f/mistralrs-core/src/models/layers.rs#L84

I would be happy to contribute it.