search

Unity-Technologies / barracuda-release

Other
564 stars 76 forks source link

[Feature Request] baracuda able to run GPT4ALL4bit and AlpacaLora4bit #322

Open elephantpanda opened 1 year ago

elephantpanda commented 1 year ago

Will Baracuda be able to run this model? Or similar models:

https://huggingface.co/Sosaka/GPT4All-7B-4bit-ggml

or

https://github.com/johnsmith0031/alpaca_lora_4bit

This would be excellent if it could. This is the way AI is heading with these small optimised models.

I have been able to run 4bit Llama model on Quadro P5000 GPU (similar to GeForce 1080) using pytorch.