About the question of derivation of merge_with_quantization in the paper

yuhuixu1993 / qa-lora

Official PyTorch implementation of QA-LoRA

MIT License

117 stars 11 forks source link

About the question of derivation of merge_with_quantization in the paper #22

Open LuletterSoul opened 10 months ago

LuletterSoul commented 10 months ago

Hello, I'm curious why AB can merge to zero point, can you give a detailed derivation process? As far as I know, LoRA can fuse AB into W, why can QA-LoRA fuse AB into qzeros?

LuletterSoul commented 10 months ago

@yuhuixu1993 Congratulations to QA-LoRA for being accepted as a poster at ICLR 2024！Any responses about my question ? It will be very helpful.