Adds Phi-3-mini support. This requires updating transformers to near latest. There hasn't been a release of transformers package since Phi-3 support was added.
Dependencies:
A commit from Friday in transformers package has an issue so updating transformers package to just before that commit. Once the issue is fixed, we can update to latest git commit of transformers and once there's a release, we can point to the latest package.
Also updating peft==0.6.2. I have checked the slicing and finetuning for Phi-2 and run the tests. We get nearly the same results as the paper
model: Phi-2
piqa: originial: 79.3, sliced@25% 74.5, recovery finetuned: 74.4
ppl on alpaca: original 2.98, sliced@25%: 3.22, recovery finetuned: 2.99
We can look at renaming the adapters: llama->llama2 and Phi-3->Phi-3-mini in a separate change. I have not tried the Phi-3-mini-128k version! We will need to figure out how to handle that too if we want to.
Adds Phi-3-mini support. This requires updating transformers to near latest. There hasn't been a release of transformers package since Phi-3 support was added.
Dependencies: A commit from Friday in transformers package has an issue so updating transformers package to just before that commit. Once the issue is fixed, we can update to latest git commit of transformers and once there's a release, we can point to the latest package. Also updating
peft==0.6.2. I have checked the slicing and finetuning for Phi-2 and run the tests. We get nearly the same results as the paperWe can look at renaming the adapters: llama->llama2 and Phi-3->Phi-3-mini in a separate change. I have not tried the Phi-3-mini-128k version! We will need to figure out how to handle that too if we want to.