Does CodeT5 continue pre-training based on T5?

salesforce / CodeT5

Home of CodeT5: Open Code LLMs for Code Understanding and Generation

https://arxiv.org/abs/2305.07922

BSD 3-Clause "New" or "Revised" License

2.71k stars 396 forks source link

Does CodeT5 continue pre-training based on T5? #18

Closed BitcoinNLPer closed 2 years ago

BitcoinNLPer commented 2 years ago

Does CodeT5 continue pre-training based on origin T5? Or did you only use the T5 architecture without using the T5 weights and conduct training from scratch?

Thanks!

yuewang-cuhk commented 2 years ago

Hi, we pre-train CodeT5 from scratch on code as we build our own code-specific tokenizer and cannot leverage the T5 weights. You can find such information in the paper:)

BitcoinNLPer commented 2 years ago

Thank u!