Resource requirements for Fine-Tuning

salesforce / CodeT5

Home of CodeT5: Open Code LLMs for Code Understanding and Generation

https://arxiv.org/abs/2305.07922

BSD 3-Clause "New" or "Revised" License

2.68k stars 394 forks source link

Resource requirements for Fine-Tuning #43

Closed Debdeep1998 closed 2 years ago

Debdeep1998 commented 2 years ago

Dear sir,

We're currently trying to finetune Code-T5 and would like to know the minimum as well as recommended hardware requirements for doing so.

yuewang-cuhk commented 2 years ago

Hi, modern GPUs with >= 16 GB would be already enough to fine-tune CodeT5-small/base models. Note that you can employ a larger gradient accumulation step to maintain the same total batch size in order to reproduce our results.