Open tsantra opened 9 months ago
@jason-dai Thank you for replying. I am trying to train Alpaca Qlora on CPU. I modified the Alpaca code for GPU to run on CPU. Its running without any error so far but it is slow. Apart from using bigdl-llm-init for speed up on CPU, would you suggest any other changes for speed up for the Alpaca code without affecting training stability? I am using default hyperparameters as set in the code.
training hyperparams batch_size: int = 128, micro_batch_size: int = 2, num_epochs: int = 3, learning_rate: float = 3e-5, # default to be 3e-5 to avoid divergence cutoff_len: int = 256, val_set_size: int = 2000, gradient_accumulation_step = batch_size/micro_batch_size = 64
Hi @tsantra , apart from using bigdl-llm-init, you may also try tuning the batchsize to smaller ones.
Hi,
I have ported the Alpaca Qlora code given for GPU example, to CPU. I am using Sapphire Rapids for training.
These are my code changes:
model = AutoModelForCausalLM.from_pretrained(
base_model,
load_in_low_bit="nf4", # According to the QLoRA paper, using "nf4" could yield better model quality than "int4"
Using default values in code:
training hyperparams batch_size: int = 128, micro_batch_size: int = 2, # default to be 2, limited by GPU memory num_epochs: int = 3, learning_rate: float = 3e-5, # default to be 3e-5 to avoid divergence cutoff_len: int = 256, val_set_size: int = 2000,
trainer = transformers.Trainer( model=model, train_dataset=train_data, eval_dataset=val_data, args=transformers.TrainingArguments( per_device_train_batch_size=micro_batch_size, gradient_accumulation_steps=gradient_accumulation_steps,
warmup_ratio=0.03,
Training has been going on without any error. However, is there any setting that I should apply for better speed in CPU? Or in general do you have any guidance for training on CPU?