search

Intelligent-Computing-Lab-Yale / TesseraQ

Apache License 2.0
16 stars 2 forks source link

Missing run_awq_llama.sh #1

Open RanchiZhao opened 1 month ago

RanchiZhao commented 1 month ago

Hi, thanks for your work. However, I cannot find the 'run_awq_llama.sh', am i missing sth?

yhhhli commented 1 month ago

Hi, we just uploaded the scripts.

RanchiZhao commented 1 month ago

Hello, I've completed the setup for AWQ according to the tutorial, but when I moved on to step two to acquire the TesseraQ model, I found that using save_fp, save_autogptq, or save_lightllm didn't work as expected.

yhhhli commented 1 month ago

Are you trying to export a quantized model? Consider following the tutorials of the original LLMC framework.

RanchiZhao commented 3 weeks ago

Sorry for the delay in replying. I noticed that when I use tesseraq for w4a16 quantization, the reconstruction loss in the last few layers is surprisingly high. For example, the second to last layer dropped from 100 to 70 after using tesseraq, but the last layer dropped from 1300 to 650, while the initial layers were only around 2. Do you have any suggestions? Does this indicate that the quantization is not going smoothly?

yhhhli commented 3 weeks ago

Hi, the reasons are twofold. (1) The block reconstruction error will accumulate across blocks. (2) The later blocks naturally have larger activation values, (if you visualize them you will find out), which means the loss would get higher than former blocks.