search

intel / auto-round

Advanced Quantization Algorithm for LLMs. This is official implementation of "Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs"
https://arxiv.org/abs/2309.05516
Apache License 2.0
172 stars 20 forks source link

cohere model support request #103

Closed MichoChan closed 3 months ago

wenhuach21 commented 4 months ago

Hello, it seems that the CoHere model is mainly tailored for RAG. Although the LLM component is expected to be supported, it's worth noting that the embedding model tends to be relatively small and computation-bound, hence it may not yield significant benefits from weight-only quantization. Please feel free to correct me. We'll prioritize exploring this if it could bring large benefits to the community.

wenhuach21 commented 4 months ago

Apologies for any confusion, just to clarify, are you referring to https://huggingface.co/CohereForAI/c4ai-command-r-v01 instead of https://huggingface.co/Cohere? We will have a test of https://huggingface.co/CohereForAI/c4ai-command-r-v01

MichoChan commented 4 months ago

Apologies for any confusion, just to clarify, are you referring to https://huggingface.co/CohereForAI/c4ai-command-r-v01 instead of https://huggingface.co/Cohere? We will have a test of https://huggingface.co/CohereForAI/c4ai-command-r-v01

yes，i mean https://huggingface.co/CohereForAI/c4ai-command-r-v01 and https://huggingface.co/CohereForAI/c4ai-command-r-plus

wenhuach21 commented 4 months ago

yes，i mean https://huggingface.co/CohereForAI/c4ai-command-r-v01 and https://huggingface.co/CohereForAI/c4ai-command-r-plus

Ok, have you already met some issues? If the LLM is "typical", AutoRound should have already supported without requiring any additional code or configuration. Anyway, we will have a test.

MichoChan commented 4 months ago

yes，i mean https://huggingface.co/CohereForAI/c4ai-command-r-v01 and https://huggingface.co/CohereForAI/c4ai-command-r-plus

Ok, have you already met some issues? If the LLM is "typical", AutoRound should have already supported without requiring any additional code or configuration. Anyway, we will have a test.

Not yet，i will have a try

wenhuach21 commented 4 months ago

I've conducted a preliminary test on https://huggingface.co/CohereForAI/c4ai-command-r-v01 and it appears that AutoRound works well. However, given our current prioritized tasks, we plan to postpone reporting the recipe until we have available compute resources.

MichoChan commented 4 months ago

thanks a lot

wenhuach21 commented 3 months ago

close this as there is no issue, feel free to reopen it.