Open sorasoras opened 4 months ago
Really awesome
Oh yep saw it! We might instead be adding LoRA+, which has similar results. Technically DoRA is already enabled in Unsloth, but just not optimized
Oh yep saw it! We might instead be adding LoRA+, which has similar results. Technically DoRA is already enabled in Unsloth, but just not optimized
dora+qlora work already? I thought you can implement this on top of current implementation of dora
@sorasoras yes but it's not that optimized - only somewhat
@danielhanchen
What are current limitations of QDoRA in unsloth? I can't get it to work with FA2. It seems to work without FA2 but on very low ctx. Should FA2 with QDoRA be supported by current version unsloth or not?
Here's a traceback of FA2 fail in case you want to take a look.
My understanding is that LoRA+ and DoRA are relatively orthogonal and likely stack.
@adamo1139 It should work QLoRA + DoRA - but not optimized - simply turn it with use_dora = True
Unsure on that exact error msg, but if you use our Colab notebooks and just set that flag, it should work
@DreamGenX Oh DoRA gets rid of the alpha scaling for LoRA entirely, and makes it learnable
@danielhanchen @adamo1139 I had the same error if and only if I used DoRA in my codebase, but I could also confirm that it does work in the colab.
After further investigation I found that the difference is that in the colab, flash-attn is not installed. (x-formers is installed though.)
I then uninstalled flash-attn from where I usually run my code, and this got rid of the error. I wonder if removing flash-attn might lead to significant slow-downs though?
No slowdowns at all!
https://www.answer.ai/posts/2024-04-26-fsdp-qdora-llama3.html
That looks awesome!