search

salesforce / CodeT5

Home of CodeT5: Open Code LLMs for Code Understanding and Generation
https://arxiv.org/abs/2305.07922
BSD 3-Clause "New" or "Revised" License
2.68k stars 394 forks source link

Multitask Pre-training #51

Closed MichaelFu1998-create closed 2 years ago

MichaelFu1998-create commented 2 years ago

Dear authors of CodeT5, Thanks for contributing such an amazing model to the community. In the paper, it is said that CodeT5 was pre-trained using multiple tasks. I'm wondering how these task were arranged, did you pre-train multiple tasks all-in-one and combine the loss or did you pre-train each task one by one. Thank you very much for your help :)

Kind regards Michael

yuewang-cuhk commented 2 years ago

Hi, we sample one task for each batch based on a balanced sampling strategy during multi-task pretraining. Please refer to here for more details.