search

allenai / scibert

A BERT model for scientific text.
https://arxiv.org/abs/1903.10676
Apache License 2.0
1.47k stars 214 forks source link

How to finetune Scibert with multiple GPUs? #106

Closed chloefresh closed 2 years ago

chloefresh commented 3 years ago

I’d like to finetune scibert in multiple GPUs, not just one. But export CUDA_DEVICE=0 in _train_allennlplocal.sh means only gpu0 is used. So where do I need to modify to utilize multiple GPUs?

fan-hd commented 2 years ago

I’d like to finetune scibert in multiple GPUs, not just one. But export CUDA_DEVICE=0 in _train_allennlplocal.sh means only gpu0 is used. So where do I need to modify to utilize multiple GPUs?

Multiple gpu finetuning has been implemented in the codes using DataParallel from torch. Maybe you can change export CUDA_DEVICE=0 to export CUDA_DEVICE=0,1,2,3 and you should change the way parsing this argument cuda_device in config files.