Open bleedingfight opened 6 months ago
Same issue
cross-ref #1118
Hi, there was a bug causing non-stopping behavior in python runtime in general. Can you please check #1118 and re-test the upcoming weekly release to verity this issue is gone? Thanks
cross-ref https://github.com/NVIDIA/TensorRT-LLM/issues/1118#issuecomment-1968045987
The main branch has been updated. Thanks.
@kaiyux thanks for your update,I tried compiling the latest code in docker,tensorrt-llm:0.9.0.dev2024022700,ON 3090 The maximum batch used when building the engine file is 8. run.py batch=1,2,4,8: My Test: | batch | 0.9.0.dev2024022700(s) | 0.9.0.dev2024020600(s) |
---|---|---|---|
1 | 4.064309024810791 | 4.487630772590637 | |
2 | 4.015249347686767 | 4.456121301651001 | |
4 | 4.471942019462586 | 4.894955468177796 | |
8 | 5.405202007293701 | 5.83393361568451 |
Hi @bleedingfight What is the actual number of generated tokens for transformers and TRT-LLM?
Although max_new_tokens
is set to 200, transformers output has 22 tokens only while TRT-LLM output has 200 tokens. When both TRT-LLM and transformers generate same number of tokens (by setting max_new_tokens=22
), TRT-LLM is 5x faster.
@amukkara In my application scenario, we need to output image descriptions for many images.So the max_new_token for our output is the same for different images. I can indeed improve performance by reducing the value of max_new_token,but this is only because the number of iterations output by the auto-regressive model has decreased. I think the optimization of TRT-llm is not deep enough.
@bleedingfight
Using higher max_new_tokens
is not an issue. But we should ensure TRT-LLM and transformers generate similar output length to make fair comparison. You can change options to generate()
like top_k
, min_new_tokens
and replicate them in transformers script. Once the output token counts are similar, TRT-LLM should be much faster than transformers.
@amukkara How to maintain consistent output between trt-llm and transformers,I use the generate interface in Transformers with max_new_tokens=200,topk=50 and same prompt,The output results of transformers and trt-llm are different.I want to know how NVIDIA officially ensures the correctness of the TRT model output, but it seems difficult for developers themselves to know how NVIDIA ensures it.
@amukkara How to maintain consistent output between trt-llm and transformers,I use the generate interface in Transformers with max_new_tokens=200,topk=50 and same prompt,The output results of transformers and trt-llm are different.I want to know how NVIDIA officially ensures the correctness of the TRT model output, but it seems difficult for developers themselves to know how NVIDIA ensures it.
same question. We test on qwen-vl-chat. Have you solved this problem yet? Looking forward to any suggestions.
System Info
Who can help?
No response
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction
Expected behavior
output not much faster then transformers!
actual behavior
[02/21/2024-11:36:57] [TRT-LLM] [I] TensorRT vision encoder latency: 0.0076348876953125 sec [02/21/2024-11:36:57] [TRT-LLM] [I] TensorRT-LLM LLM latency: 4.899357304573059 sec [02/21/2024-11:36:57] [TRT-LLM] [I] Generate latency: 4.9331824231147765 sec
additional notes
I only modified a part of the code that is not very related to the algorithm,code
llava.py
:run_demo.sh
pipeline with transformers(llava_demo.py):
Although the above code may take around 10 seconds to test a single inference on my machine,Perhaps it was an accidental result, as my previous testing took about 5 seconds.Although there is significant fluctuation in a single test, it can stabilize to around 5 seconds after a large number of tests。Why is llava trt-llm not much faster than transformers?![image](https://github.com/NVIDIA/TensorRT-LLM/assets/11495161/78c7b69b-ba95-4c06-a441-d582554b5505)
During operation, the GPU utilization rate did not reach 100%