search

99991 / pygguf

GGUF parser in Python
MIT License
16 stars 3 forks source link

Support for Q4_0 and other formats. #3

Open Azure-Tang opened 1 month ago

Azure-Tang commented 1 month ago

Hi, I notice some dequant format, like Q4_0 Q5_0, are not supported yet. Will you support these formats?

99991 commented 1 month ago

There usually are other formats with better perplexity and lower file size than Q4_0 and Q5_0. Do you really need them?

99991 commented 1 month ago

I have not heard back from you, but I have now implemented Q4_0 and Q5_0 anyway. Could you test whether it works for your use case?

Azure-Tang commented 1 month ago

sorry i didn't check this message box, I would test it in a few days and give you feedback. Thanks!

ExtReMLapin commented 6 days ago

Should be fixed in https://github.com/99991/pygguf/pull/4

99991 commented 6 days ago

Should be fixed in #4

What does that pull request have to do with this issue?

ExtReMLapin commented 6 days ago

At the end of the day (not literally), gguf package will handle dequantization, so all quantization type will be supported.