Open bclarkson-code opened 2 months ago
Currently, only single-gpu computation is supported. Multi-gpu support with NCCL should be added.
We can start with simple data parallelism but I would also like to add ZeRO so we can scale to more GPU's in the future
Currently, only single-gpu computation is supported. Multi-gpu support with NCCL should be added.