In the attention operations of the LLM, text tokens must use the causal mask to predict the next text tokens for the answer. However, the image features are also masked by a causal mask rather than by a bidirectional mask in the attention operations of the LLM, even though the bidirectional mask allows access to more visual information than the causal mask. Is there a particular reason for this making?
Yebulabula
commented
3 months ago
Question
In the attention operations of the LLM, text tokens must use the causal mask to predict the next text tokens for the answer. However, the image features are also masked by a causal mask rather than by a bidirectional mask in the attention operations of the LLM, even though the bidirectional mask allows access to more visual information than the causal mask. Is there a particular reason for this making?