Closed zeeshansayyed closed 3 years ago
@zeeshansayyed I think the acceleration brought by DDP is very limited since the batch size of the training set is 1/n (num of devices you used, so this allows you to train with large batch on small GPUs like 1080ti) of that on single device. But practically, it can slightly speed up the training. I'm not sure why DDP slowed down your training, could you give me more details?
Thanks @yzhangcs To be honest, I don't know what the issue was. A reboot fixed it. What was happening was that the GPU usage of all GPUs would max out at 100% and it would get stuck there. I think the issue is not with your code but either some kind of race condition someplace else. I will close this for now and will open it again if I am able to reproduce the issue.
Hello, thanks for sharing this project. I was trying to train the parser using DDP as is shown in the ReadMe. Unfortunately, this is twice as slow compared to using a single GPU. On a single GPU, it takes about 25 seconds whereas when I use two GPUs (installed on the same machine), it just seems to pause at the start of a new epoch and hence takes about 58 seconds.
I understand that this could possible not be an issue with your code, but was wondering if you could provide any pointers. I am using exactly the same style of command that is given in the ReadMe.