Closed NooneOnlyOne closed 2 years ago
We found that you tested different window sizes for local self-attention, and I want to know when the window is Full, what the n_mha_win_size should be set to
For example, you should set n_mha_win_size to -1 in the config file.
n_mha_win_size
The code are from here.
If mha_win_size >1, local attention is used, otherwise global attention is used.
Thank you for the reply
We found that you tested different window sizes for local self-attention, and I want to know when the window is Full, what the n_mha_win_size should be set to