search

ml-explore / mlx-examples

Examples in the MLX framework
MIT License
5.52k stars 797 forks source link

TinyLlama LoRA Example #138

Open fakerybakery opened 6 months ago

fakerybakery commented 6 months ago

Hi, I saw this example and was wondering if it might be possible to train TinyLlama with LoRA. I haven't been able to figure out how to convert the weights. Might it be possible to provide an example? Thank you! cc: @awni

awni commented 6 months ago

It's a great idea, we'd love to extend the LoRA example to more models. You can see how we convert TinyLlama here

fakerybakery commented 6 months ago

Ah, nice - before I was trying to convert it with the Phi script :). So does LoRA w/ MLX supports TinyLlama?

awni commented 6 months ago

There are a couple things missing in the LoRA example to work with TinyLlama (namely RoPE without traditional).

fakerybakery commented 6 months ago

Hi, Thank you so much! Is there a timeline for adding LoRA support for TinyLlama, now that the 3T version is released? Thank you!

awni commented 6 months ago

I got it working yesterday by using the conversion script in the llama example with a couple of small changes in the lora example. I think it makes sense to update the lora example to support downloading HF models directly. I can't give an exact ETA yet, but shouldn't take that long to implement.

fakerybakery commented 6 months ago

Thanks! Also is there a way to convert MLX -> HF?