Could this HyenaDNA model be used for a pure language task? Of course with some changes, such as a tokenizer for language! And maybe some other things ? Which other things would that be?
If this can be done, than that would give an enormous advantage of being able to work with a giant context size while still having an acceptable (or even a very good) performance for training and inference! Am I correct here?
Also I saw a mention of the HyenaDNA model being able of in-context learning, which is a very important prerequisite of such a model!
I have not read the paper, but could you show a table with a comparison between the pros and cons of a standard normal transformer and a HyenaDNA model?
But my main question is:
Could this HyenaDNA model be used for a pure language task? And how exactly to go about actually implementing that? What would be gained in comparison with using a conventional Transformer for that?
Thank you for this amazing development! What a time to be alive!
Happy New Year for you and the entire team of HyenaDNA!
exnx
commented
7 months ago
Could this HyenaDNA model be used for a pure language task? Of course with some changes, such as a tokenizer for language! And maybe some other things ? Which other things would that be?
If this can be done, than that would give an enormous advantage of being able to work with a giant context size while still having an acceptable (or even a very good) performance for training and inference! Am I correct here?
Also I saw a mention of the HyenaDNA model being able of in-context learning, which is a very important prerequisite of such a model!
I have not read the paper, but could you show a table with a comparison between the pros and cons of a standard normal transformer and a HyenaDNA model?
But my main question is: Could this HyenaDNA model be used for a pure language task? And how exactly to go about actually implementing that? What would be gained in comparison with using a conventional Transformer for that?
Thank you for this amazing development! What a time to be alive!
Happy New Year for you and the entire team of HyenaDNA!