I am trying to do a 30B finetune on 2x3090 using data parallel and the process will always crash before 3 steps is completed. I run the finetune script in a screen session on a remote computer, and the screen session is gone when i reestablish the SSH connection after the crash.
I am trying to do a 30B finetune on 2x3090 using data parallel and the process will always crash before 3 steps is completed. I run the finetune script in a screen session on a remote computer, and the screen session is gone when i reestablish the SSH connection after the crash.
This is the command I use to start finetune
Has anyone else gotten the same crashing problem?