use_fast_fftconv generates error

[FeelingFatigued]

Hi. I've tried to set use_fast_fftconv as True in H3 module, but it generates einops error saying as follows.

einops.EinopsError:  Error while processing rearrange-reduction pattern "b 1 h -> b h".
 Input tensor shape: torch.Size([1, 2, 768]). Additional info: {}.
 Shape mismatch, 2 != 1

rearrange() in line 189 of h3.py generates that error. What should I change to make it run with use_fast_fftconv option?

[DanFu09]

From the shapes, it looks like you're running generative inference. The FFTConv option is generally only used for training, where you need to process a full sequence at once. For inference, you can use the generation script: https://github.com/HazyResearch/H3/blob/main/examples/generate_text_h3.py .

To get a feel for H3 during training, you can see the safari repo: https://github.com/HazyResearch/safari, in particular this doc: https://github.com/HazyResearch/safari/blob/main/experiments.md (the section on the Pile has relevant commands/configs for training).

[FeelingFatigued]

Thanks for your reply!!