Add more informative checks for ZeRO incompatibility.

EleutherAI / gpt-neox

An implementation of model parallel autoregressive transformers on GPUs, based on the Megatron and DeepSpeed libraries

https://www.eleuther.ai/

Apache License 2.0

6.96k stars 1.02k forks source link

Closed AI-WAIFU closed 2 months ago

AI-WAIFU commented 2 months ago

-Adds clear messages when trying to use ZeRO 2/3 pipeline or model parallel. -Update is_pipe_parallel comment