Closed QLutz closed 3 weeks ago
Hey @QLutz , I suspect it may be related to #2099. Can you try to run TGI with --cuda-graphs 0
see if you still see the hang?
I had the same problem, and was able to solve it by trying --cuda-graphs 0
method. This obviously caused major performance problems, but it was at least a better option than being broken.
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.
System Info
OS version: Linux Model being used (curl 127.0.0.1:8080/info | jq): TheBloke/Nous-Hermes-2-Mixtral-8x7B-DPO-AWQ Hardware used (GPUs, how many, on which cloud) (nvidia-smi): 1xL40S The current version being used: 2.0.4
Information
Tasks
Reproduction
Launch TGI using
max_total_tokens
=max_batch_prefill_tokens
=16384;max_input_length
=16383;quantize
=awq. After making a few hundred requests, the pod returns empty packets and only a few seconds after the request a has been made. Monitoring reveals thattgi_queue_size
increases steadily but does not ever go down.Expected behavior
No stutters.