search

mlabonne / llm-course

Course to get into Large Language Models (LLMs) with roadmaps and Colab notebooks.
https://mlabonne.github.io/blog/
Apache License 2.0
36.84k stars 3.86k forks source link

i-Quants in AutoQuant? #66

Open GameOverFlowChart opened 4 months ago

GameOverFlowChart commented 4 months ago

Would it be possible to support i-Quants in AutoQuant or are they more demanding to quantize?

mlabonne commented 4 months ago

Can't you already create iquants by providing the right name? https://github.com/ggerganov/llama.cpp/blob/04976db7a819fcf8bfefbfc09a3344210b79dd27/gguf-py/gguf/constants.py#L811

GameOverFlowChart commented 3 months ago

Oh right in this case this should be added to the list of the names that is shown in the notebook, at least one of them as an example (so that you see at first glance that there is no underscore between I and Q.