Closed twaka closed 1 year ago
After digging into the issue, I noticed that there are possible cause of bug in ParallelGpt.cc if it helps.
https://github.com/NVIDIA/FasterTransformer/blob/bc077a9078b99244c69cfe0dc44af86fd974bc71/src/fastertransformer/models/multi_gpu_gpt/ParallelGpt.cc#L1002-L1009 Since prompt is added, it might need to account for prompt lengths like GPTJ or GPTNeoX doing.
https://github.com/NVIDIA/FasterTransformer/blob/bc077a9078b99244c69cfe0dc44af86fd974bc71/src/fastertransformer/models/multi_gpu_gpt/ParallelGpt.cc#L1334-L1336 I think prefix_soft_prompt_embedding is not defined as parameter in docs. It makes max_prefix_soft_prompt_length always 0.
This issue is fixed in FT v5.2 release and ft backend 1.3 release. Please try again.
Thank you very much. v5.2 solves the issue.
Description
When using GPT model and T4 GPU with triton server, setting request_prompt_lengths causes to leak the previous inference's response. In the second request, its response contains the tokens of first request's response in the lasts. test.py is pasted below.
Reproduced Steps
prepare GPT model by following the doc https://github.com/triton-inference-server/fastertransformer_backend/blob/main/docs/gpt_guide.md
now in docker
edit config.pbtxt's model_checkpoint_path and run server
send requests with request_prompt_lengths by following python script test.py