microsoft / DeepSpeed-MII

MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.
Apache License 2.0
1.91k stars 175 forks source link

DeepSpeed with Phi-3-mini-128K-instruct does not generate `<|endoftext|>` token #529

Open shubhanshu786 opened 1 month ago

shubhanshu786 commented 1 month ago

I am using Phi-3-mini-128K-instruct model for inference via DeepSpeed for high throughput and facing issue of model not generating <|endoftext|> during the generation time. For all my requests it generates all max_new_tokens size output and generation stops abruptly in the middle of the sentence due to token generation limit.

It generates <|end|> token in the text, which I am using as of now to limit the output but in some cases, it does not even generate <|end|> token as well. Tested same model with vLLM engine and huggingface as well, and there it is able to generate <|endoftext|> token and stops token generation, so clearly issue is with DeepSpeed-MII only.

Expected Behaviour: During the inference <|endoftext|> (eos_token) should be generated and output generation should be stoped.

[2024-09-27 10:03:56,258] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)

DeepSpeed C++/CUDA extension op report

NOTE: Ops not installed will be just-in-time (JIT) compiled at runtime if needed. Op compatibility means that your system meet the required dependencies to JIT install the op.

JIT compiled ops requires ninja ninja .................. [OKAY]

op name ................ installed .. compatible

async_io ............... [NO] ....... [OKAY] fused_adam ............. [NO] ....... [OKAY] cpu_adam ............... [NO] ....... [OKAY] cpu_adagrad ............ [NO] ....... [OKAY] cpu_lion ............... [NO] ....... [OKAY] [WARNING] Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH evoformer_attn ......... [NO] ....... [NO] fp_quantizer ........... [NO] ....... [OKAY] fused_lamb ............. [NO] ....... [OKAY] fused_lion ............. [NO] ....... [OKAY] /home/ec2-user/anaconda3/envs/pytorch_p310/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to dlopen' /home/ec2-user/anaconda3/envs/pytorch_p310/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference todlclose' /home/ec2-user/anaconda3/envs/pytorch_p310/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference to dlerror' /home/ec2-user/anaconda3/envs/pytorch_p310/compiler_compat/ld: /usr/local/cuda-12.1/lib64/libcufile.so: undefined reference todlsym' collect2: error: ld returned 1 exit status gds .................... [NO] ....... [NO] inference_core_ops ..... [NO] ....... [OKAY] cutlass_ops ............ [NO] ....... [OKAY] transformer_inference .. [NO] ....... [OKAY] quantizer .............. [NO] ....... [OKAY] ragged_device_ops ...... [NO] ....... [OKAY] ragged_ops ............. [NO] ....... [OKAY] random_ltd ............. [NO] ....... [OKAY] [WARNING] sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.4 [WARNING] using untested triton version (3.0.0), only 1.0.0 is known to be compatible sparse_attn ............ [NO] ....... [NO] spatial_inference ...... [NO] ....... [OKAY] transformer ............ [NO] ....... [OKAY] stochastic_transformer . [NO] ....... [OKAY]

DeepSpeed general environment info: torch install path ............... ['/home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/torch'] torch version .................... 2.4.1+cu121 deepspeed install path ........... ['/home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/deepspeed'] deepspeed info ................... 0.15.1, unknown, unknown torch cuda version ............... 12.1 torch hip version ................ None nvcc version ..................... 12.1 deepspeed wheel compiled w. ...... torch 2.4, cuda 12.1 shared memory (/dev/shm) size .... 7.72 GB

loadams commented 3 weeks ago

Thanks for reporting this @shubhanshu786 - I'll try to take a look.