[REQ] Add support for phi-3 mini model

sozercan / aikit

🏗️ Fine-tune, build, and deploy open-source LLMs easily!

https://sozercan.github.io/aikit/

MIT License

358 stars 27 forks source link

[REQ] Add support for phi-3 mini model #209

Closed surenderssm closed 4 months ago

surenderssm commented 4 months ago

What kind of request is this?

New feature

What is your request or suggestion?

I would like to add a support for phi-3 mini model.

https://huggingface.co/microsoft/Phi-3-mini-128k-instruct

I am very new to this ecosystem, however I have a strong desire to learn and contribute in this space, pointers or references will be enough for me to get started. Let me know your thoughts!

Are you willing to submit PRs to contribute to this feature request?

[X] Yes, I am willing to implement it.

sozercan commented 4 months ago

@surenderssm thanks for opening an issue! I just added the phi 3 3.7b model. see https://sozercan.github.io/aikit/premade-models

this is not the 128k version but the 4k one. if you are interested in contributing, it'll be similar to #210 but for 128k gguf file. I didn't see the gguf for the 128k one at the time

sozercan commented 4 months ago

Looks like llama.cpp doesn't support 128k yet due to longrope scaling type: https://github.com/ggerganov/llama.cpp/issues/6849

sozercan commented 3 months ago

Looks like https://github.com/ggerganov/llama.cpp/pull/7225 added support for 128k version. aikit v0.9.0 includes the llama.cpp version that has this feature.