search

elixir-nx / bumblebee

Pre-trained Neural Network models in Axon (+ 🤗 Models integration)
Apache License 2.0
1.33k stars 96 forks source link

Support PHI-3 #376

Closed fire closed 3 months ago

fire commented 3 months ago

Phi-3-medium-128k-instruct-Q8_0.gguf

Want to test https://huggingface.co/bartowski/Phi-3-medium-128k-instruct-GGUF

The Phi-3-Medium-128K-Instruct is a 14B parameters, lightweight, state-of-the-art open model trained with the Phi-3 datasets that includes both synthetic data and the filtered publicly available websites data with a focus on high-quality and reasoning dense properties. The model belongs to the Phi-3 family with the Medium version in two variants 4k and 128K which is the context length (in tokens) that it can support.

jonatanklosko commented 3 months ago

We support Phi-3, but the repo you linked is in GGUF quantized format, which we do not support. You should be able to load from the official checkpoint microsoft/Phi-3-medium-128k-instruct.

jonatanklosko commented 3 months ago

Ah, and you need to be using Bumblebee main branch.