search

chu-tianxiang / exl2-for-all

EXL2 quantization generalized to other models.
8 stars 2 forks source link

make_q_matrix(): incompatible function arguments #3

Open mirix opened 10 months ago

mirix commented 10 months ago

I copied and pasted the inference example, but I receive the following error message:

Traceback (most recent call last):
  File "/home/emoman/Work/rag/exl2-for-all/test_exl2forall.py", line 19, in <module>
    quant_model = Exl2ForCausalLM.from_quantized("turboderp/Llama2-7B-exl2", revision="2.5bpw")
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/emoman/Work/rag/exl2-for-all/model.py", line 151, in from_quantized
    model = post_init(model)
            ^^^^^^^^^^^^^^^^
  File "/home/emoman/Work/rag/exl2-for-all/utils.py", line 170, in post_init
    submodule.post_init(temp_dq=model.device_tensors[device])
  File "/home/emoman/Work/rag/exl2-for-all/qlinear.py", line 80, in post_init
    self.q_handle = ext_make_q_matrix(self.q_tensors, temp_dq)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/emoman/Work/rag/exl2-for-all/qlinear.py", line 33, in ext_make_q_matrix
    return ext_c.make_q_matrix(w["q_weight"], w["q_perm"], w["q_invperm"],
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: make_q_matrix(): incompatible function arguments. The following argument types are supported:
    1. (arg0: torch.Tensor, arg1: torch.Tensor, arg2: torch.Tensor, arg3: torch.Tensor, arg4: torch.Tensor, arg5: torch.Tensor, arg6: torch.Tensor, arg7: torch.Tensor, arg8: torch.Tensor, arg9: torch.Tensor, arg10: torch.Tensor) -> int
chu-tianxiang commented 10 months ago

Sorry. Exllamav2 changed the function arguments to allow for different group size in v0.10.0 while this repo is not updated. I'll look into it.

chu-tianxiang commented 9 months ago

Code updated to be compatible with current version of exllamav2.

mirix commented 9 months ago

It works now! Thanks a million!