Open runzeer opened 3 years ago
when using the DataParallel, the memory cost in each GPU is not equal. So if convenient, could you change the mode to DDP?
Yep, we are aware that alternative multi-GPU methods like DDP have some advantages over DataParallel, but switching to DDP will introduce non-trivial complexity, and we do not currently have plans to incorporate it, although we are not ruling it out for the future.
The main runscript should automatically detect the number of GPUs available and run with DataParallel. Are you encountering any issues doing so?