Ubuntu 22.04
Triton image: nvcr.io/nvidia/tritonserver:24.06-trtllm-python-py3 and the version of trtllm-backend is 0.10.0
Model: qwen2-7b-instruct
Who can help?
No response
Information
[ ] The official example scripts
[ ] My own modified scripts
Tasks
[ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[ ] My own task or dataset (give details below)
Reproduction
launch a qwen2-7b-instruct in a container with image nvcr.io/nvidia/tritonserver:24.06-trtllm-python-py3, and then test ensemble and bls model.
1 ensemble model test
1.1 there are no characters between two sentences in input text
1.2 there are some sapce or '\n' between two sentences in input text, the semantics of input text is not changing
2 bls model test
2.1 there are no characters between two sentences in input text
2.2 there are some sapce or '\n' between two sentences in input text, the semantics of input text is not changing
Expected behavior
the result of 1.1 and 1.2 is same
the result of 2.1 and 2.2 is same
System Info
Ubuntu 22.04 Triton image: nvcr.io/nvidia/tritonserver:24.06-trtllm-python-py3 and the version of trtllm-backend is 0.10.0 Model: qwen2-7b-instruct
Who can help?
No response
Information
Tasks
examplesfolder (such as GLUE/SQuAD, ...)Reproduction
launch a qwen2-7b-instruct in a container with image nvcr.io/nvidia/tritonserver:24.06-trtllm-python-py3, and then test ensemble and bls model.
1 ensemble model test 1.1 there are no characters between two sentences in input text
1.2 there are some sapce or '\n' between two sentences in input text, the semantics of input text is not changing
2 bls model test 2.1 there are no characters between two sentences in input text
2.2 there are some sapce or '\n' between two sentences in input text, the semantics of input text is not changing
Expected behavior
the result of 1.1 and 1.2 is same the result of 2.1 and 2.2 is same
actual behavior
the result of 2.1 and 2.2 is not same
additional notes
no