search

triton-inference-server / tensorrtllm_backend

The Triton TensorRT-LLM Backend
Apache License 2.0
706 stars 106 forks source link

how to process batch #65

Open zhanglv0209 opened 1 year ago

zhanglv0209 commented 1 year ago
image

image

I hope there are two results，exp： "hello" ，"你好”，but one result , I do not know how to process batch .

xiaoFine commented 1 year ago

the preprocess only encode the first element in input text preprocess.py

so u need to modify the serving code

zhanglv0209 commented 1 year ago

the preprocess only encode the first element is input text preprocess.py

so u need to modify the serving code

How to modify it? Can you give me a demo? Thank you very much

zhanglv0209 commented 1 year ago

the preprocess only encode the first element in input text preprocess.py

so u need to modify the serving code

Hello, I tried to make changes to the file you mentioned, but found that the changes weren't correct. Could you provide me with some editing tips? I would greatly appreciate it.

hubble-bubble commented 12 months ago

the preprocess only encode the first element in input text preprocess.py so u need to modify the serving code

Hello, I tried to make changes to the file you mentioned, but found that the changes weren't correct. Could you provide me with some editing tips? I would greatly appreciate it.

Have you solved this question？