Closed absol13 closed 1 year ago
@honglu2875
@absol13 Would #979 fix the issue? It was my oversight that I didn't realize data
could be None
. It could be possible that it hangs because some process errored out due to this bug.
@honglu2875 I don't know the exact reason. Anyway, I will try distributed training after #979 is merged and let you know this bug is solved. Thanks.
I have reproduced this issue and confirmed that #979 does not fix it.
I tried to set up neox from an empty conda env again, and used my own config (pythia-1b) but made sure
On the main branch it just errored out on the first training step instead of hanging. Applying #979, it can train normally (watched it for about 10 training steps).
Will take a look at other items in OP's config later. But hope this helps narrow down the problem.
After conferring with @honglu2875 we discovered that I was failing to apply the fix to both nodes. @absol13 It should now work for you on main
Now it works correctly. Thanks to your fast support.
Describe the bug Hello, I found distributed training with the setting
"model-parallel-size": >1
hangs. This situation appears in the source with the PR #958 is merged, and it did not appear in older sources or with the setting"model-parallel-size": 1
at all.To Reproduce I share my config file below for reproduction.
Expected behavior Training should proceed further.
Proposed solution
Screenshots Specifically, training does not proceed at this point:
Also, I report logs from nvidia-smi command.
In the normal training procedure, GPU power usage reaches near its capacity, but I observed that it is far from its capacity which seems to mean the hanging process.
Environment (please complete the following information):
Configs: I share my config file for training to help reproducing this situation.