still fail to apply on llama-13b

AIoT-MLSys-Lab / SVD-LLM

Official Code for "SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression"

https://arxiv.org/abs/2403.07378

Apache License 2.0

68 stars 6 forks source link

still fail to apply on llama-13b #7

Closed 33answer33 closed 2 months ago

33answer33 commented 2 months ago

Hi, thank you for your reply. But I still get the same problem as mentioned before. Traceback (most recent call last): File "/home/xxx/SVD-LLM/SVDLLM_new.py", line 193, in whitening scaling_matrix_inv = torch.linalg.inv(scaling_diag_matrix) torch._C._LinAlgError: linalg.inv: The diagonal element 6940 is zero, the inversion could not be completed because the input matrix is singular. " My python environment is built on requirements.txt. And I run the code on 2 3090 GPUs

tuidan commented 2 months ago

Hi, thank you for your feedback.

It seems that the calculation of inversion also needs to be executed in a double()precision.

I have updated the code. Now the LLaMA-13B runs successfully on my side as follows:

python SVDLLM.py --model jeffwan/llama-13b-hf --step 1 --save_path ./

python SVDLLM.py --model_path /local/scratch0/jeffwan_llama_13b_hf_whitening_only_0.8.pt --step 4

33answer33 commented 2 months ago

I still fail to run it on 3090.Anyway, thank you so much, perhaps I should try using different GPUs..