Closed zhiwei-dong closed 11 months ago
Yeah, we have not yet supported LET for LLaMA-2-70B.
Get
Here's another question, I tried to turn off some smooth, but it seems that multi-card is not taken into account when quantizing 70b models
Thanks for your feedback. I have fixed this bug.
get
I set batchsize = 1
Why is 40GB of graphics memory used during the training phase. But in the end, is it just consuming such a large amount of graphics memory?
Why did I use A800/80G to run the compression of llama65B, which resulted in overflow of graphics memory? As mentioned in the paper, 40GB of graphics memory is enough to compress 65B? @zhiwei-dong @ChenMnZ
There seems to be an issue with the OmniQuant Llama2-70b model. The problem arises from the mismatch of scales shape and weight in the GQA algorithm.