Order of hyena operation

[sylee0124]

The order of operation described in the hyena paper was conv in time domain first then frequency domain. But the implementation in this repo does conv in frequency domain first then time domain. Am I missing something?

https://github.com/HazyResearch/safari/blob/main/src/models/sequence/hyena.py#L339

[Zymrael]

This small change was to accommodate checkpoints of models trained on a different codebase, the order of these two ops does not make a huge difference as long as you perform both (see how there is an additional gating at the output, so you can view the sequence of ops as pre-gate followed by the regular conv-gate order described in the paper).

[sylee0124]

Thanks for the clarification!

[sylee0124]

Sorry but I have one more question. Doesn't current implementation reduces the number of time domain conv? For example, in the paper order-2 hyena (which is corresponds to H3) has two frequency domain conv and two time domain conv. But for current implementation, the order-2 hyena has two frequency domain conv and a single conv time domain conv. Should I just think that order-2 hyena in current code is equivalent to order-1 hyena in the paper?

[Zymrael]

that's correct there is one less long convolution in the current implementation. The number of total time-domain convolutions is still 2 if you count the short input convolution (which replaces the shift SSM in H3). The changes were made to carry out fair ablations (in terms of total FLOPs and model depth) between H3 and order-2 Hyenas.

[sylee0124]

The number of total time-domain convolutions is still 2 if you count the short input convolution (which replaces the shift SSM in H3).

Which means that order-2 hyena with short conv + long conv is used for all evaluations in the paper?

The changes were made to carry out fair ablations (in terms of total FLOPs and model depth) between H3 and order-2 Hyenas.

When just comparing model quality without consideration of FLOPs, is order-2 hyena with two long conv better than a short-conv and a long-conv? or is this short-conv essential for model quality?

[Zymrael]

Which means that order-2 hyena with short conv + long conv is used for all evaluations in the paper?

Yes, anywhere we state Hyena-2 in the manuscript. There are some experiments with other orders (e.g. wikitext).

When just comparing model quality without consideration of FLOPs, is order-2 hyena with two long conv better than a short-conv and a long-conv? or is this short-conv essential for model quality?

Higher orders can be more performant depending on the task, but you should always keep at least one short convolution.