search

bigscience-workshop / Megatron-DeepSpeed

Ongoing research training transformer language models at scale, including: BERT & GPT-2
Other
1.31k stars 213 forks source link

How to inference GPT2 with DeepSpeed? #346

Closed cdj0311 closed 2 years ago

cdj0311 commented 2 years ago

Hi, I only found bloom model inference in scripts, but how to inference GPT2 with deepspeed which trained with Megatron-DeepSpeed? I convert megatron-deepspeed model to transformers with tools/convert_checkpoint/deepspeed_to_transformers.py, then modify checkpoint_json = {"type": "Megatron","checkpoints": file_list,"version": 1.0} to inference , but I get an error as follows:

/home/anaconda3/envs/py37/lib/python3.7/site-packages/deepspeed/module_inject/replace_module.py", line 789, in replace_transformer_layer checkpoint = checkpoint_dict['checkpoints'] TypeError: 'MegatronSDLoader' object is not subscriptable

mayank31398 commented 2 years ago

Hf models are readily compatible with deepspeed. Deepspeed.initialize wraps any nn.module class object