search

MAGICS-LAB / DNABERT_2

[ICLR 2024] DNABERT-2: Efficient Foundation Model and Benchmark for Multi-Species Genome
Apache License 2.0
212 stars 49 forks source link

Flash Attention 2 #64

Closed jacobfulano closed 5 months ago

jacobfulano commented 5 months ago

Hi @Zhihan1996,

You might also be interested in this PR which updates MosaicBERT to FlashAttention 2 and removes all the issues with the custom Triton FlashAttention + ALiBi implementation.