Open Rishabhg71 opened 8 months ago
Hi @Rishabhg71 I suspect that running with mp_size=2,
in the script and --num_gpus 4
is causing a problem. Please make sure these values match.
Additionally, for Llama-2 models we suggest using the latest DeepSpeed-MII to run inference.
Hello @mrwyattii ! I am facing the same error [2024-02-12 22:12:33,072] [ERROR] [launch.py:321:sigkill_handler] ... exits with return code = -9
. For reference, I have followed the DS Inference HF example , and successfully ran inference using the Falco-40B model with 8 A100 GPUs. Then I switched from Falcon-40B to Falcon-180B, and this error above occurred. From the metrics graph, the GPUs are not even started. It happens right after the model was successfully loaded with message Loading checkpoint shards: 100%
.
Could you provided any insights into this issue? Thank you!
Describe the bug
I was trying to run an inference with DeepSpeed on the Llama model, but when I ran
deepspeed --num_gpus 4 script.py
, the process terminated automatically after loading the checkpoint shards, without providing any additional information. Also, when runningnvidia-smi
, it appears that the model wasn’t even loaded, and no process was created, or it was created for a minimal amount of time.To Reproduce
deepspeed --num_gpus 4 script.py
from transformers import LlamaForCausalLM, AutoTokenizer, AutoModel
import deepspeed import torch
Specify the model name
model_name = "meta-llama/Llama-2-13b-hf"
Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_name)
Load the model
model = LlamaForCausalLM.from_pretrained(modelname, token="hf")
model = AutoModel.from_pretrained(modelname, token="hf")
Initialize the DeepSpeed-Inference engine
ds_engine = deepspeed.init_inference(model, mp_size=2, dtype=torch.half) model = ds_engine.module output = model('Input String')
System info (please complete the following information):