search

HazyResearch / hyena-dna

Official implementation for HyenaDNA, a long-range genomic foundation model built with Hyena
https://arxiv.org/abs/2306.15794
Apache License 2.0
574 stars 82 forks source link

Flash Attention 2 #9

Closed dbrami closed 1 year ago

dbrami commented 1 year ago

Hi team Hazy Research, It's just a matter of time before you get this question but is HyennaDNA going to use Flash Attention 2 vs 1? The improvements listed on repo for v2 seem pretty significant but v. 1 is linked in HyennaDNA. I also see that you work with Flash Attention team based on author/ contributor list so probably won't be long until we see this change...

exnx commented 1 year ago

Hello! So to be clear, HyenaDNA itself does not use attention (or FlashAttention).

However, we do compare with FlashAttention in this repo. In future works, we may compare with FlashAttn2 going forward, but for this repo, it won't fundamentally change any HyenaDNA performance any way :)