Question regarding the time taken for training and what type of compute that's been used

[haresh121]

Hi, Can you please answer the following questions

1. How much time is taken for the DPLM to get trained on the datasets mentioned ?
2. What are the memory requirements and on what type of GPU that you've performed the training on ?

[wxy-nlp]

Hi @haresh121, Sorry for the late reply. The time and memory cost during training is as below: 1) DPLM 150M:

• batch size on one GPU: 10240 tokens
• gradient accumulation: 2
• number of GPUs: 16
• total batch size: 320K tokens
• CUDA memory cost on one GPU: ~40G using config 'dplm_150m_stage2' costs ~22 hours for 100k updates on 16 A100 GPUs. 2) DPLM 650M:
• batch size on one GPU: 8192 tokens
• gradient accumulation: 4
• number of GPUs: 32
• total batch size: 1M tokens
• CUDA memory cost on one GPU: ~70G using config 'dplm 650m stage2' costs ~3days for 100k updates on 32 A100 GPUs.