facebookresearch / xformers

Hackable and optimized Transformers building blocks, supporting a composable construction.

https://facebookresearch.github.io/xformers/

Other

8.69k stars 621 forks source link

Customization of BlockDiagonalMask or LowerTriangularMask #1147

Open kc334 opened 1 week ago

kc334 commented 1 week ago

❓ Questions and Help

I am trying to achieve a variant of block mask. This is how it look. Snipaste_2024-11-13_16-03-37 in this ex, sequence length = 16, block_size =4. Is it possible to achieve it with current codes of BlockDiagonalMask or LowerTriangularMask