Why is the calculation result of tensorrt-llm version llava1.5 different from the output of HF?

[bleedingfight]

System Info

• tensorrt-llm:0.9.0.dev2024022700
• GPU:L40S
• tensorrtl-llm docker
• driver:535.129.03

Who can help?

No response

Information

• [X] The official example scripts
• [ ] My own modified scripts

Tasks

• [X] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• [ ] My own task or dataset (give details below)

Reproduction

1. Train the llava model using official llava code
2. Convert the official llava model to the huggingface llava model
3. Convert huggingface llava model to Tensorrt-llm benchmark(mmbench_cn): official llava as thruth,hf llava 99%(The output results of the hf model are close to 99.9% compared to the official model)，llava-trt 93%。（The output of hf-lava in the following statement is taken as the true value） I randomly found an example where the output of tensorrt is different from the output of hf。trt-llava:B,hf-llava:A。 I tried the following centralized methods, but none of them solved the problem:
4. Is it the calculation error of the clip model that caused the final result to be incorrect（no）?
   1. Hugginfface uses float32 to calculate the output result of visual_tower, and then converts it to float16 input of language_model through mmprojector.Language model's inputs_embeds not same,I use hf's inputs_embeds for tensorrt-llm,the output is still B not A of llava-hf。 language_model input is same: hf output: llava-trt:

Expected behavior

llm with same input will be same output： '\n人们可以使用工程设计过程来解决问题。该过程中的一步是测试潜在解决方案是否符合设计要求。\n下面的段落描述了如何使用工程设计过程来测试解决问题的方案。阅读段落，然 后回答下面的问题。\n\nDevin是一名机械工程师，他正在设计一个记录温度、降水和风速的天气站。这个天气站将被用于一个最高记录温度为40摄氏度的城镇。Devin希望确保即使在异常炎热 的天气下，天气站也能正常工作。\n因此，他将一个室内测试室设置为50摄氏度，湿度低且无风。他将天气站留在测试室过夜。第二天，他检查天气站在50摄氏度下经过24小时后是否显示准确 的测量结果。\n图：一个天气站。\n以下哪项是Devin的测试可能显示的？\nA. 当温度为50°C时天气站是否能正常工作\nB. 当有风时天气站的工作情况如何\n请直接回答选项字母。

actual behavior

trt-llm:output(B)[[ 1, 319, 13563, 1546, 263, 12758, 1404, 322, 385, 23116.......] (Pdb) output_ids[0,0,:2] tensor([ 1, 319], device='cuda:0', dtype=torch.int32) (Pdb) output_ids[0,0,input_lengths[0]:] tensor([350, 2, 2, ..., 2, 2, 2], device='cuda:0', dtype=torch.int32)

additional notes

hf:output(A)[tensor([319, 2], device='cuda:0')]

[byshiue]

Due to different kernel selection and kernel implementation, it often generates different results. Unless there are obvious accuracy regression, we think it is reasonable.

[bleedingfight]

@byshiue I have seen a significant decrease in the accuracy of the output results of TRT on my test set. I would like to know how you have determined that the output of TRT is reasonable.

[byshiue]

We use mmlu and summarization task to evaluate. Could you try reproducing the accuracy on public model and public example, sharing your reproduced steps and let us be easier to reproduce your issue?

[bleedingfight]

@byshiue Thank you very much for your reply. I'm sorry for the delayed reply. In the past few days, I have been trying to provide a Docker and minimum reproduction code. For the convenience of reproduction, I have upgraded to the latest TRT-LLM(0.11.0.dev2024051400). Now my previous code cannot produce results. look like this #1632

[byshiue]

temperature 0.0 is not a valid number in current TensorRT-LLM. Please use greedy search and don't set temperature directly, or set a very small temperature like 1e-6.

[bleedingfight]

@byshiue oI modified temperature=1e-6 according to your statement, but I found that errors occur at all times except for the first inference that produces output，self.tokenizer.batch_decode(output_ids[0, :, input_lengths[0] :])will produce error: like this #1299 。but my trt-version=0.11.

[byshiue]

Thank you for the try. Could you share the full end to end steps to reproduce your issue? (how to convert ckpt, building engine and run the example)

[bleedingfight]

@byshiue here

[byshiue]

Could you share the end to end steps to reproduce your issue? It is hard for me to understand how to use the scripts you share.

[bleedingfight]

@byshiue readme.md in zip.I just start webserver for llava-trt to process the result

[Jaxx4Fun]

temperature 0.0 is not a valid number in current TensorRT-LLM. Please use greedy search and don't set temperature directly, or set a very small temperature like 1e-6.

So how to enable greedy search?