How can I use the same prompt to produce the same text output as vllm

microsoft / DeepSpeed-MII

MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.

Apache License 2.0

1.85k stars 174 forks source link

How can I use the same prompt to produce the same text output as vllm #482

Open Greatpanc opened 4 months ago

Greatpanc commented 4 months ago

I used the same model and the same input, and found that the results obtained by the two inference frameworks vllm and deepspeed mii were inconsistent. I need to configure deepspeed mijin x. I am using the default configuration of vllm.