Open emillykkejensen opened 4 months ago
Hey @emillykkejensen, unfortunately our min supported architecture at the moment is Ampere due to the flash attention dependency. Please see system requirements here: https://github.com/predibase/lorax?tab=readme-ov-file#requirements
Fair enough. However, one could argue that the point of qlora among other things, is to serve on smaller (older and cheeper) GPU's that don't support ampere? Is there anything in the making, or?
Yes, we have plans to move our attention computation over to the FlashInfer project, which is working on support for Volta and Turning GPUs. So hopefully that will address the issue.
Sounds good 😊 I'm sure you are already aware, but in the off case your not, I can see that there is a fix in TGI? However it seems they simply fix it by loading the full model?
Is it fair to assume that this should now work given this PR https://github.com/predibase/lorax/issues/440 is merged ? With latest versions, I seem to be able to get past FA 2 errors but seems to run into different issue https://github.com/predibase/lorax/issues/535
System Info
Information
Tasks
Reproduction
I'm simply trying to run mistralai/Mistral-7B-v0.1 with 4-bit quantization on my T4 with bitsandbytes-nf4! However it errors with 'Mistral model requires flash attn v2'?
Expected behavior
The model to load and run!?