drprojects
commented
6 months ago Hi @Charlie839242, thanks for your interest in the project !
That is correct, using the ddp.yaml config should allow you to do multi-gpu training. For more information, please refer to the documentation from the lightning-hydra template and lightning on which this project is based.
Two remarks though:
- our work focuses on efficient and lightweight models. Hence, multi-GPU is not really our prioritized research direction, so we have not tested it
- our models use GraphNorm and not BatchNorm. You will have to investigate whether the GraphNorm implementation we use properly interfaces with Lightning multi-GPU. I have doubts about it... An easy fix would be to replace the GraphNorms by BatchNorms in your model config and train from scratch, at the cost of a (small) drop in performance in our experiments
PS: if you like or use this project, please give it a ⭐, it means a lot to us !
Charlie839242
Hi Damien,
Thanks for your great paper and awesome project! I am not very sure about one point: simply changing the gpu.yaml to ddp.yaml will make the model train in DDP model, is this correct? And is synchronization of batchnorm supported in this case? Thanks for you reply!