search

HazyResearch / H3

Language Modeling with the H3 State Space Model
Apache License 2.0
511 stars 53 forks source link

Full version of H3 model? #29

Open FeelingFatigued opened 11 months ago

FeelingFatigued commented 11 months ago

Hello,

This repository does not seem to have the complete H3 described in the paper. (e.g., sequence length must be <= 8192 and I couldn't find state passing algorithm in the source)

If it is possible, would you please push the most recent version of the H3 model described in the paper?

DanFu09 commented 10 months ago

We have a new kernel coming out very soon that obviates that algorithm and scales up to 4M - will post here when we release it!

FeelingFatigued commented 10 months ago

I noticed the new FlashFFTConv repo was released recently. I've been trying to use that in this H3 model but I am not sure if I am changing the code correctly.

If it is possible, could you update H3 repo with FlashFFTConv? or could you tell me where do I need to change?

Thanks in advnace.

DanFu09 commented 10 months ago

Ack’ing that I’ve seen this, will try to get it in this week or next!

On Fri, Nov 24, 2023 at 3:07 AM FeelingFatigued @.***> wrote:

I noticed the new FlashFFTConv repo was released recently. I've been trying to use that in this H3 model but I am not sure if I am changing the code correctly.

If it is possible, could you update H3 repo with FlashFFTConv? or could you tell me where do I need to change?

Thanks in advnace.

— Reply to this email directly, view it on GitHub https://github.com/HazyResearch/H3/issues/29#issuecomment-1825285275, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABDDIIUOZSRQUODR7NVUOUTYGBIUBAVCNFSM6AAAAAA6VRHSM2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRVGI4DKMRXGU . You are receiving this because you commented.Message ID: @.***>

FeelingFatigued commented 9 months ago

Sorry but can I know when will the FlashFFTConv-enabled H3 be available?

Thanks for your help.