AI4Bharat / IndicBERT

Pretraining, fine-tuning and evaluation scripts for IndicBERT-v2 and IndicXTREME
https://ai4bharat.iitm.ac.in/language-understanding
MIT License
73 stars 13 forks source link

Add Flash Attention 2 Support #5

Open rajveer43 opened 1 year ago

rajveer43 commented 1 year ago

Feature request

Flash Attention 2 is a library that provides attention operation kernels for faster and more memory efficient inference and training: https://github.com/Dao-AILab/flash-attention

269989726-1395f962-26ca-4728-a8d0-085792295c28