Closed Hukongtao closed 2 months ago
For attention_mask_type, I tried both causal and bidirectional, but the results didn't make any difference. https://github.com/NVIDIA/TensorRT-LLM/blob/3d56a445e8ebf888e78be638faf6beec0a78f3c2/tensorrt_llm/models/qwen/model.py#L133
For attention_mask_type, I tried both causal and bidirectional, but the results didn't make any difference.
What is the difference between AttentionMaskType.causal and AttentionMaskType.bidirectional? Why use different types but the hidden_state is the same?
Hi I solved my problem by change the true to false: https://github.com/NVIDIA/TensorRT-LLM/blob/3d56a445e8ebf888e78be638faf6beec0a78f3c2/cpp/tensorrt_llm/plugins/gptAttentionCommon/gptAttentionCommon.cpp#L1439
As the title describes, I slightly modified the QWenAttention. Before:
After:
self.is_causal = False
I use tensorrt-llm to accelerate the modified Qwen model, but there will be a problem of inconsistent numerical accuracy. I want to know if there is a way to support the modified QWenAttention.