Open bnuzhanyu opened 4 months ago
Hi @bnuzhanyu, you got the "TensorrtLLM output" from the source code of "Tensorrt-llm Print input image and output embedding:", right? And how do you got the "Qwen-VL-Chat ModelScope" results? If the ViT engine and the input are same, the results are expected the same.
Hi @bnuzhanyu, you got the "TensorrtLLM output" from the source code of "Tensorrt-llm Print input image and output embedding:", right? And how do you got the "Qwen-VL-Chat ModelScope" results? If the ViT engine and the input are same, the results are expected the same.
# line 565
images = self.visual.encode(images)
to get the "Qwen-VL-Chat ModelScope" results
any update?
Hi @calico-niko @bnuzhanyu The ViT is offloaded to TRT, and the fp32 accuracy of it on TRT9.3 is alined with Pytorch. And you can also change the version of TRT from 9.3 to 10.x, the fp16 accuracy of it on TRT10.x is fine.
Hi @calico-niko @bnuzhanyu The ViT is offloaded to TRT, and the fp32 accuracy of it on TRT9.3 is alined with Pytorch. And you can also change the version of TRT from 9.3 to 10.x, the fp16 accuracy of it on TRT10.x is fine.
I updated trt to 10.0.1 , but got the same diff. I think the results vary a lot.
vit by trt: tensor([[[ 1.1836, -0.6230, 0.5547, ..., -0.2083, -1.2617, -3.3867], [ 0.4189, -0.3127, -0.5732, ..., 0.8232, 1.0723, 0.8164], [-0.4614, -0.0329, 0.7266, ..., 0.3604, -1.1826, -0.0151], ..., [-0.3298, 1.5420, 1.1074, ..., 1.4434, -0.7012, -2.1191], [ 0.2686, -0.4331, -2.0234, ..., -0.1218, -0.9346, -0.0122], [-1.3047, 0.8560, -2.2266, ..., -0.5923, 1.6758, 0.3738]]], device='cuda:0', dtype=torch.float16)
vit by visual.encode(images): tensor([[ 1.2646, -0.6035, 0.6665, ..., -0.1432, -1.2197, -3.3691], [ 0.2174, -0.3809, -1.2285, ..., 1.6143, 0.8530, 0.3674], [-0.4709, -0.0422, 0.7241, ..., 0.3362, -1.1611, -0.0426], ..., [-0.3340, 1.5625, 1.0889, ..., 1.5625, -0.6138, -2.0215], [ 0.2142, -0.3896, -2.0020, ..., -0.0946, -0.9102, 0.0299], [-1.3281, 0.9053, -2.2617, ..., -0.5645, 1.7812, 0.4031]], device='cuda:0', dtype=torch.float16)
@sunnyqgg Do you have any other suggestion?
Hi @hezeli123 , the diffs are smaller compared with TRT 9.x, does the current ViT diffs have a big impact on the final results? If so, you can try to run ViT with FP32 precision.
The current ViT diffs have a big impact which results in many bad cases. I run ViT with FP32 precision now.
OK, if you have strong desire to use FP16, I'll continue to look at this issue, if not, this issue will have a lower priority.
Problem
For same input image , I get different output of the visual embedding, and this could make the result a little bit worse than original model.
env
tensorrt-llm 0.9.0, GPU: A10 model: https://modelscope.cn/models/qwen/Qwen-VL-Chat/summary
Qwen-VL-Chat build command
Tensorrt-llm Print input image and output embedding:
Different output using exmaples/qwenvl/pics/demo.jpeg
TensorrtLLM output
Qwen-VL-Chat ModelScope
The input image is same, but the output embeddings are not close.