Added quantization support for huggingface models

vanna-ai / vanna

🤖 Chat with your SQL database 📊. Accurate Text-to-SQL Generation via LLMs using RAG 🔄.

https://vanna.ai/docs/

MIT License

9.12k stars 673 forks source link

Added quantization support for huggingface models #509

Closed SMAntony closed 2 weeks ago

SMAntony commented 2 weeks ago

Hi Guys, Here's my pull request discussed in #502 . I have added quantization support for huggingface models and have changed a variable 'model_name' to 'model_name_or_path' as transformers can load models from checkpoint files in local directory.