search

artidoro / qlora

QLoRA: Efficient Finetuning of Quantized LLMs
https://arxiv.org/abs/2305.14314
MIT License
9.92k stars 817 forks source link

Compatibility with Deepspeed, Fairscale, or Torch zero-redundancy optimizer #6

Open SparkJiao opened 1 year ago

SparkJiao commented 1 year ago

Wonderful work!

May I know the compatibility with ZeRO mechanism? E.g., Torch redundancy optimizer, deepspeed zero-1 to zero-3, and fairscale FSDP. Becaused I noticed that QLoRA relies on particularly implemented optimizer.

If the optimizer is not compabitible with the tools mentioned above, can I use only 4-bit tuning and lora with zero mechanism? Will this cause more memory cost?

Thanks very much!

Best

alibabadoufu commented 1 year ago

Wonderful work!

May I know the compatibility with ZeRO mechanism? E.g., Torch redundancy optimizer, deepspeed zero-1 to zero-3, and fairscale FSDP. Becaused I noticed that QLoRA relies on particularly implemented optimizer.

If the optimizer is not compabitible with the tools mentioned above, can I use only 4-bit tuning and lora with zero mechanism? Will this cause more memory cost?

Thanks very much!

Best

I did try this a few days ago but I think currently deepspeed still cannot support 4 bit training (backpropagation)