search

ml-explore / mlx-examples

Examples in the MLX framework
MIT License
5.8k stars 824 forks source link

Phi1.5 LoRA example #135

Open harryjulian opened 8 months ago

harryjulian commented 8 months ago

I've managed to get Phi1.5 fine-tuning with LoRA on my machine using a combination of the LoRA and Phi2 examples. Would it be worth me opening a PR to add this to the examples? If so, where would it fit?

awni commented 8 months ago

Good question! We probably shouldn't have a separate example for this. But think it would be great to try and generalize the LoRA example to support multiple models. Right now it supports llama (v1?) and mistral, but it would be great to add phi-* support for it as well.

shchahrykovich commented 8 months ago

@harryjulian can you share your code, I think I can help to adopt it

harryjulian commented 8 months ago

Here's the scripts I cobbled together.