Closed gkielian closed 3 months ago
This should fix the compatibility of torchrun with train.py
Simpler fix found in latest pr: https://github.com/ReaLLMASIC/nanoGPT/pull/170
Fixing this line plus adding gradient_accumulation_steps flag 16 does the trick
This should fix the compatibility of torchrun with train.py