Closed PenutChen closed 1 week ago
cc @SunMarc
The correct workaround is to replace num_blocks
in this code with -1
, but I'm not sure if this is the correct behavior.
# transformers/integrations/ggml.py
def dequantize_q6_k(data):
block_size = GGML_BLOCK_SIZES["Q6_K"]
num_blocks = len(data) // block_size
data_f16 = np.frombuffer(data, dtype=np.float16).reshape(-1, block_size // 2)
data_u8 = np.frombuffer(data, dtype=np.uint8).reshape(-1, block_size)
data_i8 = np.frombuffer(data, dtype=np.int8).reshape(-1, block_size)
scales = data_f16[:, -1].reshape(-1, 1).astype(np.float32)
Hey @PenutChen thanks for opening the issue ! I tried your snippet on the main branch of transformers and on v4.42.3, and everything looks fine ! I suggest you to clear your cache and try it again. Also, which version of numpy are you using ? Maybe this is an issue with the 2.0 version was released recently.
@SunMarc Thanks for the reply! I upgraded the numpy version to 1.26.4, but I still get the same error. After checking all my dependencies, I found that my gguf
was installed from the source of the llama.cpp repo. I changed the version to the PyPI one, and it works!
Thanks for investigating ! Hopefully, for the next release of gguf, we won't have the issue you experienced.
The latest release of the gguf package is from Dec 13, 2023, but the gguf source still updates frequently. There are some incompatible settings between them. For anyone experiencing this issue, try the following commands:
pip install gguf==0.6.0 "numpy<2.0" --force-reinstall
Hi @SunMarc, just a reminder that gguf-py has been updated to 0.9.1 recently. There might be some issues with this version. If I find anything new, I will reopen this issue.
Hi @PenutChen, thanks for the warning ! It looks like we indeed have failing tests on side. We get the same error you experienced. I will reopen the issue =)
downgrading to gguf==0.6.0 solved it for me. Thanks for @PenutChen hint.
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
solved by #32298
System Info
transformers==4.42.3 torch==2.3.0
Who can help?
No response
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction
The example usage from doc:
Expected behavior
Produce the following error: