Open 152334H opened 9 months ago
--num-gpus is implemented by sharding each expert layer across GPUs, i.e. expert parallelism
--num-gpus
this is probably not advisable for local experimentation, especially on batch size 1 -- where EP only adds communication overhead to no speed benefit vs naive model/pipeline parallel.
Good suggestions, I am working on other parallelism method. Also, contribution is welcomed.
--num-gpus
is implemented by sharding each expert layer across GPUs, i.e. expert parallelismthis is probably not advisable for local experimentation, especially on batch size 1 -- where EP only adds communication overhead to no speed benefit vs naive model/pipeline parallel.