Closed dhcode-cpp closed 10 months ago
I found Xformers API use block-diagonal-mask that can make cross-batch independently
Ref: https://facebookresearch.github.io/xformers/components/ops.html
CLASSxformers.ops.fmha.attn_bias.BlockDiagonalMask
That input is [1, batch1_len+batch2_len+batch3_len, H, K]
My data batch size = 3, windows_size = 3, the input like is
I have two questions when I debugging mistral model;
First, 3 batch sequences would be flat as a one sequence [5, 7, 2] -> tensor like [5+7+2, 1]?
Second, If first things is true, how do we calculate attention?
We print Q/K/V shape before mistral/model.py:
and print string as following(the layer number is 2, n_kv_head =4 and n_head = 4):
------------------ 0 cur_layer_id : 0 q: torch.Size([1, 17, 4, 128]) k: torch.Size([1, 17, 4, 128]) v: torch.Size([1, 17, 4, 128]) ------------------ 1 cur_layer_id : 1 q: torch.Size([1, 17, 4, 128]) k: torch.Size([1, 17, 4, 128]) v: torch.Size([1, 17, 4, 128]) ------------------ 0 cur_layer_id : 0 q: torch.Size([1, 3, 4, 128]) k: torch.Size([1, 9, 4, 128]) v: torch.Size([1, 9, 4, 128]) ------------------ 1 cur_layer_id : 1 q: torch.Size([1, 3, 4, 128]) k: torch.Size([1, 9, 4, 128]) v: torch.Size([1, 9, 4, 128])
Mistral is an impressive work, and I'm excited to hear your response. Thank you very much!