search

alipay / PainlessInferenceAcceleration

Creative Commons Attribution 4.0 International
268 stars 17 forks source link

modeling_qwen attention not use multi branch position ids & attention_mask #27

Open snippetzero opened 1 month ago

snippetzero commented 1 month ago

I reviewed the code of modeling_qwen.py, and I noticed that, within the lookahead process, the draft_ids matched from the TrieTree are such that the attention_mask and position ids associated with these draft_ids are not being utilized in the attention mechanism. This, I believe, might be an implementation error. Could you please point out where my understanding is incorrect?

YunjiaXi commented 3 days ago

I believe there are issues with modeling_qwen.py. The content generated by Qwen differs significantly when using lookahead compared to when it is not used. With lookahead, the output text becomes much longer and exhibits incoherence. After fixing the attention mask issue you mentioned, the text became much more coherent, but there are still substantial differences in the content generated. I think there are still some unresolved problems in this code.