Closed tareqmahmood closed 1 month ago
According to the following: https://github.com/JF-D/Proteus/blob/0bb4cb1c977e3c0626e18c3ba7b0bfdbc463780c/examples/megatron_gpt.py#L317
megatron strategy expects pp_deg to be 1. Is there any reason behind this?
megatron
pp_deg
The megatron strategy represents using megatron tensor parallelism, together with zero strategy. If you want to use megatron with pipeline parallelism, just use -ps pp and sepcify -mp-deg.
zero
-ps pp
-mp-deg
According to the following: https://github.com/JF-D/Proteus/blob/0bb4cb1c977e3c0626e18c3ba7b0bfdbc463780c/examples/megatron_gpt.py#L317
megatron
strategy expectspp_deg
to be 1. Is there any reason behind this?