mallorbc
commented
10 months ago I think ultimately the core of the issue is that regular Lora can be on CPU. Accelerate then puts the model where it needs to go. QLora has to be on a GPU and then:
ValueError: You can't train a model that has been loaded in 8-bit precision on a different device than the one you're training on. Make sure you loaded the model on the correct device using for example device_map={'':torch.cuda.current_device()}you're training on. Make sure you loaded the model on the correct device using for example `device_map={'':torch.cuda.current_device() or device_map={'':torch.xpu.current_device()}Maybe a way to pass a QLora config would be ideal?
younesbelkada
hengjiUSTC
I am using TRL and PEFT to finetune models. You can see the code I am using here
When finetuning Llama 7B, I use Lora because with A100s I can fit the model in this format and its a bit faster than QLora.
When finetuning 70B, I must use Qlora.
When finetuning with Lora, the code will automatically utilize all available GPUs with DDP. I do not need to use accelerate, though I can and when I do, it works. I imagine there is some stuff going on behind the scenes that is using accelerate
When using QLora, the model does not work, but the behavior changes based on whether or not I am using accelerate explicitly.
When not explicitly using Qlora, the model is loaded onto both GPUs, but one GPU is idle and not ever used. Due to this, the model training never progresses.
We can see in this photo the model is loaded on both but only training is occurring on GPU 0.
When I use accelerate explicitly, I get the following error