What's the different between Instruction Tuning to Align with Natural Language Instructions and Finetune Using Your Own Data? - Githubissues

salesforce / CodeT5

Home of CodeT5: Open Code LLMs for Code Understanding and Generation

https://arxiv.org/abs/2305.07922

BSD 3-Clause "New" or "Revised" License

2.65k stars 391 forks source link

What's the different between Instruction Tuning to Align with Natural Language Instructions and Finetune Using Your Own Data? #130

Open JustinZou1 opened 11 months ago

JustinZou1 commented 11 months ago

What's the different between Instruction Tuning to Align with Natural Language Instructions and Finetune Using Your Own Data?

yuewang-cuhk commented 11 months ago

Hi there, these two are very similar and you can view the instruction tuning as a specific use case of finetuning on your own data. In the instruction tuning example, we showcase how to finetune CodeT5+ 16B on A100-40G with deepspeed and mix precision training, while the finetuning on your own data example is more for smaller models (220M/770M). So if you want to finetune CodeT5+ 2B/6B/16B, the instruction tuning example would be better example to follow.