search

allenai / open-instruct

Apache License 2.0
1.1k stars 145 forks source link

QLoRA is very slow #109

Closed SihengLi99 closed 5 months ago

SihengLi99 commented 5 months ago

Hi,

I find that qlora is much slower than lora with the same lora config. Could you please explain the reason for this?

Best

SihengLi99 commented 5 months ago

I found that the time cost of QLoRa is five times that of LoRa.

hamishivi commented 5 months ago

Hi, that sounds odd - at the very least, QLoRa should take a similar time as LoRA with similar config. Did you raise the batch size with QLoRa (which should be doable thanks to the lower memory footprint)?

SihengLi99 commented 5 months ago

Thanks for the quick reply! I did not raise the batch size to guarantee a fair comparison.

Hamish Ivison @.***> 于2024年2月8日周四 00:45写道：

Hi, that sounds odd - at the very least, QLoRa should take a similar time as LoRA with similar config. Did you raise the batch size with QLoRa (which should be doable thanks to the lower memory footprint)?

— Reply to this email directly, view it on GitHub https://github.com/allenai/open-instruct/issues/109#issuecomment-1932454386, or unsubscribe https://github.com/notifications/unsubscribe-auth/AR4KH4MVETPYQQQHLMUTV7DYSOVQ7AVCNFSM6AAAAABC5YRQNOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZSGQ2TIMZYGY . You are receiving this because you authored the thread.Message ID: @.***>