search

microsoft / DeepSpeed

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
https://www.deepspeed.ai/
Apache License 2.0
33.6k stars 3.94k forks source link

Inference with the MoE based GPT model trained by ds_pretrain_gpt_345M_MoE128.sh [BUG] #5647

Open haoranlll opened 2 weeks ago

haoranlll commented 2 weeks ago

The existing scripts failed to inference with the MoE based GPT model. The MoE policy provided in the source code https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/module_inject/containers/megatron_gpt_moe.py can't handle the MoE model at all. Is there a quick way to fix this problem?