Phi-3 small (7B) and medium (14B)

[rwl4]

It would be great to see these models work!

NotImplementedError: Unsloth: /srv/models/Phi-3-medium-4k-instruct not supported yet! Make an issue to https://github.com/unslothai/unsloth!

Done. :)

[shimmyshimmer]

We're working on it! Phi 3 medium support will most likely come out first and then small due to differing architecture.

[danielhanchen]

@rwl4 Currrently we support phi-3 mini via https://colab.research.google.com/drive/1NvkBmkHfucGO3Ve9s1NKZvMNlw5p83ym?usp=sharing and https://huggingface.co/unsloth/Phi-3-mini-4k-instruct-bnb-4bit

[joshib123]

any updates on this?

[JackCloudman]

waiting for it! So thanks. :smile:

[shimmyshimmer]

It's out!! @JackCloudman @joshib123 @rwl4

https://x.com/danielhanchen/status/1793762458437578955

Phi 3 medium and mini. Small will be supported later

[danielhanchen]

@rwl4 @JackCloudman @joshib123 We support Phi-3 Medium and Mini now! See https://github.com/unslothai/unsloth/releases/tag/May-2024 (also includes Colabs)

Small is still in the works!

Please update Unsloth for local machines. For Colab or Kaggle just refresh and restart the env!

pip uninstall unsloth -y
pip install --upgrade --force-reinstall --no-cache-dir git+https://github.com/unslothai/unsloth.git

[joshib123]

Hi @danielhanchen, thanks for the update. But, when I tried to finetune the Phi-3-medium the training loss goes from 1.80 to 0 after first step. Wondering if there is a bug somewhere in the code? PS: the same code worked for other models (such as Llama-3-8b).

[danielhanchen]

@joshib123 I don't think there's a bug - that probably means ur learning rate is too high

[anakin87]

@danielhanchen thanks for the impressive work! Any news on phi3 small?

[danielhanchen]

@anakin87 No sorry - Small is a vastly different architecture :(