Zymrael
commented
6 months ago Thank you for the kind words. I'll use a notation close to the paper to make it easier to connect the dots, and set the number of projections to 3. In the first version of Hyena, you will have a different matrix for each channel. To visualize H(u) for a target channel you'll need to materialize the following matrices:
- T_v: toeplitz matrix corresponding to the causal short 1d convolution applied to the target channel of the
vprojection - T_x1: same as above, but for a different projection
- T_x2: same as above, but for a different projection
- D_v: the L x L matrix with
vat the given channel on the diagonal. Note this is also L x L - D_x1: same as above
- D_x2: same as above
- T: Toeplitz matrix corresponding to the long implicit convolution.
Once you have these matrices, you can multiply them in the same order you would apply them to v, e.g,:
- H(u) := T_x2 D_x2 T T_x1 D_x1 T_v v
Once you have the matrix you can also check whether applying H(u) gives you the same output via a direct matrix multiply. When you generate the Toeplitz matrices with the convolutional filters, be careful to take into account padding to keep all convolutions causal.
Firstly, I would like to extend my sincere gratitude for your amazing work and the code that you have shared. It's truly impressive.
I was reading through your paper and noticed the visualization of the Hyena Matrix H(u), which I found to be particularly interesting. However, when I went through the codebase, I realized that there isn't a direct implementation for materializing this matrix.
Could you kindly guide me on how I might go about visualizing the Hyena Matrix H(u) using my own dataset? Any pointers or additional information you could provide would be greatly appreciated.