Closed intelyoungway closed 2 months ago
can you attach training logs to ease analysis?
Sure, I am asking for customer's feedback.
@intelyoungway, the attached script is also doing LoRA finetuning. Would you clarify what is the exact issue/request?
customer said they modified the original LoRA script to do finetune (see in the attached files). And the issue is, the speed is too close between LoRA and finetuning (their modified scripts), which is strange cause LoRA should be significantly faster in theory. So the solution is simple: (1) if the attached file is correct finetuning, then please provide a LoRA script optimized so that it could be significantly faster than finetuning. (2) if incorrect, then please feedback that the script that customer modified is not a correct implementation of finetuning, and I will tell customer this message and close the ticket.
@intelyoungway, Thanks for the comment. From what you said, the goal is to compare the full parameter model fine-tuning with Lora fine-tuning.
As per the original Lora paper from Microsoft, https://arxiv.org/abs/2106.09685, it's theoretically understood that full parameter and Lora fine-tuning should not yield the same performance, mainly when low ranks are used in Lora. The disparity in the number of parameters used for training is a key factor here.
From the attached script, I see you are using the same run_lora_clm.py
script with some minor modifications for both full parameters and Lora fine-tuning. If the performance is the same, the script might have an issue. It would help if you used run_clm.py
for full parameter fine-tuning and run_lora_clm.py
for Lora fine-tuning.
For me or anyone else to be able to help, I need more details, especially log files, number of parameters, etc.
Thanks for the explanation. I think this can fulfill the need. The issue should be closed now.
Feature request