Open thies1006 opened 1 year ago
I am also running into this issue. It seems that not using the meta tensor avoids this error. That being said, https://github.com/microsoft/DeepSpeed/issues/3103 is still a problem.
@Yard1 getting this issue also when not using meta tensor. Running the mosaicml/mpt-7b model.
Describe the bug Following #2547 I tried to run the model gpt-neoxt-chat-base-20b, which is a neox-20B derivative I think and I think it should work. Inference works if the model is loaded the normal way by HF model.from_pretrained function.
To Reproduce
deepspeed --num_gpus 4 inference-test.py --name togethercomputer/GPT-NeoXT-Chat-Base-20B --batch_size 1 --ds_inference --use_kernel --use_meta_tensor --checkpoint_path '/secondary/thies/gpt-neoxt-chat-base-20b/'
Traceback:
ds_report output