Right now our training notebook supports multi-gpu training with huggingface accelerate! But when we actually try to train with multi-gpu the results never perform as well as when we train with single-gpu.
I tried adjusting learning rate and batch size but this didn't seem to fix the discrepancy in results between multi-gpu and single-gpu.
Figure out what's going on here and I will be very grateful :)
PaulScotti
commented
1 year ago
Right now our training notebook supports multi-gpu training with huggingface accelerate! But when we actually try to train with multi-gpu the results never perform as well as when we train with single-gpu.
I tried adjusting learning rate and batch size but this didn't seem to fix the discrepancy in results between multi-gpu and single-gpu.
Figure out what's going on here and I will be very grateful :)