In efficiency in causal-conv1d during token generation?

[llmexperiment]

Hi All, @tridao @albertfgu Do we have to use causal-conv1d during token generation? During token generation, essentially, we can just use dot product of weights and the inputs of size 4.
We can use causal-conv1d for conv1d during token generation, but it does unnecessary computations.

[tridao]

causal-conv1d is optional, see here.

[llmexperiment]

causal-conv1d is optional, see here.

Thank you @tridao !

Here is my understanding of casual-conv, and let me know these are correct?

• My understanding is that casuall-conv is optional only for token generation? Right?
• For prefil (prompt processing), we have to use casual-conv otherwise, prompt processing latency will be large.