OOM error when executing the bash scripts/train_backward_Myx.sh

Spico197 / Humback

🐋 An unofficial implementation of Self-Alignment with Instruction Backtranslation.

https://arxiv.org/abs/2308.06259

Apache License 2.0

129 stars 8 forks source link

Open shamanez opened 10 months ago

shamanez commented 10 months ago

I am using : 8 * A100 40GB

Spico197 commented 10 months ago

Hi there. Sorry for the late response. I run the experiment with 4*A100 40GB and didn't encounter the OOM problem. Did you apply the flash-attention?

lss11005 commented 5 months ago

I also encountered this problem,I test on 4*A100 40GB, set num_nodes=1 num_gpu_per_node=4 bsz=16 is there any problems？