search

OptimalScale / LMFlow

An Extensible Toolkit for Finetuning and Inference of Large Foundation Models. Large Models for All.
https://optimalscale.github.io/LMFlow/
Apache License 2.0
8.23k stars 822 forks source link

Causal LM finetuning #786

Open harry7171 opened 5 months ago

harry7171 commented 5 months ago

Hey,

Great to see LISA implemented here.

As for the background, I am trying to finetune models with LORA other techniques on domain data but the Task i am doing is Causal LM is Next word Prediction.

Earlier I have been using DataCollator from Hugging face to convert the data into required format and use Trainer to train the model using LORA.

I have been quite curious for LISA , and wanted to do the same with LISA. I am not getting the right process to convert my text into CausalLM data type and then use that to Finetune with LISA technique.

Any assistance/examples or references would be of great help.

research4pan commented 5 months ago

Thanks for your interest in LMFlow! You may try text-only dataset format (https://optimalscale.github.io/LMFlow/examples/DATASETS.html#textonly), which is natively supported in LMFlow. Intuitively it is doing next-word prediction for the whole sequence, and LMFlow will automatically check the type and do the causal LM next word prediction.

Other code will stay the same as examples provided in README. If you encountered any further problems, please feel free to let us know 😄

harry7171 commented 5 months ago

@research4pan Thanks for the quick reply. I would definitely try it out and let know in case of any issues.

BTW just out of curiosity , I havent checked the code yet but are you using Data collator for CausalLM (DataCollatorForLanguageModeling where mlm is False) for data prep as its pretty good for trainer and Causal LM data prep.

Thanks again for the replies

research4pan commented 5 months ago

Thanks for your valuable suggestion! We use default data collator for huggingface Trainer:

But I think this is indeed a great suggestion! We welcome any contributions that help improves LMFlow. If it is convenient on your end, you may create a PR to improve the default behavior of LMFlow. We will quickly review it once it passed the test.

Or we can schedule the support this feature in our roadmaps and implement it on our end. Thanks for your advice 😄