search

rubra-ai / rubra

Open Weight, tool-calling LLMs
https://docs.rubra.ai/
Apache License 2.0
148 stars 20 forks source link

AWQ? #119

Closed themrzmaster closed 4 months ago

themrzmaster commented 4 months ago

HI! WOuld be nice to have AWQ quantization, so we can run on vLLM with a lower end GPU. Thanks for your work!

sanjay920 commented 4 months ago

hey @themrzmaster what model are you looking for awq quants? i can get that for you

themrzmaster commented 4 months ago

70b would be nice @sanjay920

sanjay920 commented 4 months ago

@themrzmaster https://huggingface.co/rubra-ai/Meta-Llama-3-70B-Instruct-AWQ

Let me know if you have any problems with it. If it looks good feel free to close this issue.

themrzmaster commented 4 months ago

Worked! Thank you!! Any plan on releasing the dataset?